THE HOPF ALGEBRA OF DIAGONAL RECTANGULATIONS 



SHIRLEY LAW AND NATHAN READING 

Abstract. We define and study a combinatorial Hopf algebra dRec with basis 
elements indexed by diagonal rectangulations of a square. This Hopf algebra 
provides an intrinsic combinatorial realization of the Hopf algebra tBax of 
twisted Baxter permutations, which previously had only been described ex- 
trinsically as a Hopf subalgcbra of the Malvcnuto-Reutenauer Hopf algebra of 
permutations. We describe the natural lattice structure on diagonal rectan- 
gulations, analogous to the Tamari lattice on triangulations, and observe that 
diagonal rectangulations index the vertices of a polytopc analogous to the asso- 
ciahedron. We give an explicit bijection between twisted Baxter permutations 
and the better-known Baxter permutations, and describe the resulting Hopf 
algebra structure on Baxter permutations. 
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1. Introduction 

Baxter permutations arose through the work of Baxter [6] on pairs of commuting 
functions /, g : [0,1] — > [0,1]. Interest in Baxter permutations increased when a 
pleasant formula was obtained [10] for the Baxter number B(n), the number of 
Baxter permutations in S n : 

s MT)Trr§(ro(T)(::;> 

Baxter permutations also became an important example of generalized notions of 
pattern avoidance, for example in [9, 34]. 

The Baxter numbers made a surprising appearance in an algebraic context in [29] . 
There, a lattice-theoretic construction produced an infinite family of Hopf subal- 
gebras of the Malvenuto-Reutenauer [24, 25] Hopf algebra MR of permutations. 
A prominent member of this infinite family is the Hopf algebra tBax, with basis 
elements indexed by twisted Baxter permutations, which were computed in [29] to 
be counted by B(n) for n < 15. West [35] later proved, using generating trees, 
that twisted Baxter permutations are counted by B(n) for all n. Other prominent 
members of the infinite family include the Hopf algebra NSym of noncommutative 
symmetric functions [14] and the Loday-Ronco Hopf algebra LR of planar binary 
trees [22]. Just as NSym is the intersection of two anti- isomorphic copies of LR 
inside MR, the Hopf algebra tBax is the smallest member of the infinite family 
containing the two anti-isomorphic copies of LR. 

Each of NSym and LR has a basis indexed by fundamental combinatorial objects 
(subsets and planar binary trees respectively), and the Hopf algebra operations 
(product and coproduct) are described intrinsically in terms of these objects. By 
contrast, the operations in the Hopf algebra tBax are described by the construction 
in [29] only extrinsically. That is, the operations are described only with reference 
to the embedding of tBax as a Hopf subalgebra of MR. The starting point of this 
paper is an effort to provide tBax with an intrinsic description in terms of objects 
counted by B(n). A natural choice of objects is a certain class of rectangulations 
(decompositions of a square into rectangles) shown by Ackerman, Barequet, and 
Pinter [1, Lemma 5.2] to be counted by B(n). These rectangulations, which we call 
diagonal rectangulations, are closely related to an earlier construction by Dulucq 
and Guibert [11] of certain "twin" pairs of planar binary trees. They are also closely 
related to the mosaic floorplans [21, 37] in the literature on VLSI circuit design. 
(See Remark 5.3.) 

The fundamental tool in this paper is a surjective map p from permutations 
to diagonal rectangulations. This map plays a role analogous to the map from 
permutations to trees which is central to the construction of LR. In fact, p is 
combinatorially very closely related to the map to trees. We show that the fibers 
of p are the congruence classes of a lattice congruence on the weak order, and show 
that the congruence in question coincides with the congruence used in [29] to define 
tBax as a Hopf subalgebra of MR. From there, we conclude that p restricts to 
a bijection between twisted Baxter permutations and diagonal rectangulations. In 
particular, tBax can be realized as a Hopf algebra dRec with basis elements indexed 
by diagonal rectangulations. The product and coproduct in dRec are shown to have 
natural descriptions in terms of the combinatorics of diagonal rectangulations. 
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The twisted Baxter permutations come with a natural lattice structure: They 
are ordered as an induced subposet of the weak order, and this subposet is isomor- 
phic to the quotient of the weak order modulo the lattice congruence mentioned 
above. This lattice can be interpreted as a lattice on diagonal rectangulations, anal- 
ogous to the well-known Tamari lattice [32] on triangulations of a convex polygon. 
We characterize this lattice in terms of certain local moves which we call pivots, 
analogous to diagonal flips on triangulations. The Hasse diagram of the lattice of 
diagonal rectangulations, thought of as an undirected graph, is analogous to the 
1-skclcton of the associahedron. This graph, which was also considered in [1], is 
the 1-skeleton of a convex polytope analogous to the associahedron. Specifically, 
taking the Minkowski sum of two realizations of the associahedron, we obtain a 
polytope whose vertices are indexed by diagonal rectangulations and whose edges 
correspond to pivots. 

The map p is also instrumental in providing an explicit (but recursive) bijection 
between Baxter permutations and twisted Baxter permutations. The bijection is 
the restriction of the downward projection map associated to a certain lattice con- 
gruence on the weak order, and its inverse is the restriction of the corresponding 
upward projection map. Using the bijection, we construct a Hopf algebra Bax in- 
dexed by Baxter permutations, isomorphic to tBax and dRec, and having a simple 
description analogous to the original definition of tBax. Also as a consequence of 
the bijection, we show that the lattice of diagonal rectangulations is isomorphic not 
only to the subposet of the weak order induced by twisted Baxter permutations, 
but also to the subposet of the weak order induced by Baxter permutations. A 
priori it is not clear that the latter is even a lattice. 

The constructions presented here are informed by previous research on objects 
counted by the Baxter number, including in particular [1, 11, 12], but our proofs do 
not depend on any of these previous results. One part of our argument (the proof 
of Lemma 8.4) is, however, borrowed from [12]. For the most part, our proofs fall 
naturally out of the lattice-theoretic approach to Hopf algebras found in [29], and 
out of detailed combinatorial/polytopal investigations, found in [30], of maps from 
permutations to triangulations. 

In a paper [15] that appeared on the arXiv just after the present paper, Giraudo 
rediscovers the Hopf algebra tBax from [29] and studies tBax via twin binary trees 
(see Remark 6.6) and Baxter permutations. Among the results of [15] that are not 
in the present paper are a multiplicative basis for Bax, freeness, cofreeness, self- 
duality, a bidendriform structure, and primitive elements. Among the results of the 
present paper that are not in [15] are the intrinsic description of the coproduct on 
dRec, the intrinsic description of the product on dRec in terms of the defining basis, 
a complete description of cover relations in the lattice of diagonal rectangulations, 
the polytope construction, and the direct bijection between Baxter permutations 
and twisted Baxter permutations. 

2. The Hopf algebra MR of permutations 

In this section, we describe the Malvenuto-Reutenauer Hopf algebra MR, quoting 
results of [24, 25]. For more details on Hopf algebras in general, see [26, 31]. 

2.1. Product and coproduct. Let K be a field. For each integer n > 0, let K[5 n ] 
be an (n!)-dimcnsional vector space over K with basis indexed by the elements of 
the symmetric group S n . For notational convenience, we replace each basis element 
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with the permutation indexing it. Thus the elements of K[«SVi] are represented as 
K-linear combinations of permutations in S n . The Malvenuto-Reutenauer Hopf 
algebra of permutations is the graded vector space KfS'oo] = © n > K[5 n ], endowed 
with a product and coproduct that we now define. 

Let p, q and n be nonnegative integers with p + q = n. Let x = x\x 2 • • • x p e S p 
and y = y\y 2 ---y q € S q . A shifted shuffle of x and y is a permutation z = 
Z\ ■ ■ ■ z n G S n such that 

(i) the subsequence of z consisting of the values 1,2, ... ,p equals the sequence 
x\X2 ■ ■ ■ x p , and 

(ii) the subsequence of z consisting of the values p+ l,p + 2, . . . ,n equals the 
sequence y x + p, y 2 + p, ■ ■ ■ , y q + P- 
Said another way, to construct a shifted shuffle of x and y, one first "shifts" the 
entries of y up by p, and then "shuffles" the sequences x\, X2, ■ ■ ■ , x p and y\ + 
p, y 2 + p, ■ ■ ■ , y q + p together, preserving the relative order of elements within each 
sequence. Because of the shifting, this process is not symmetric in x and y. There 
are ( n ) shifted shuffles of x and y. 




The product x »s y is the sum of all of the shifted shuffles of x and y. (That is, 
a linear combination with all coefficients equal to one.) Thus, for example, 

21 » s 21 = 2143 + 2413 + 2431 + 4213 + 4231 + 4321. 
21 » s 132 = 21354 + 23154 + 23514 + 23541 

+ 32154 + 32514 + 32541 + 35214 + 35241 + 35421. 

We can think of the product as a map of graded vector spaces, from K[S' 00 ] ® K^oo] 
to K[5oo], taking a basis element x®y to x»gy. This is a graded product on K[5 f oc ], in 
the sense that, for each p and q, it restricts to a map from K[S p ] ®K{S q ] to 

The standardization of a sequence oi, ai, ■ ■ ■ , a-k of distinct integers is the unique 
permutation x € Sk such that each pair i ^ j in [k] has x% < Xj if and only if aj < aj . 
The notation std(di, a 2 , . . . , a^) stands for the standardization of a\, a 2 , . . . , a*;. For 
example, std(7, 3, 2, 11, 5) = 42153. 

The coproduct A s in K[5oo] maps x = x x x 2 • • • x n £ S n to 



The notations x\, . . . , x and x n +i, . . . , x n both stand for the empty sequence, so 
std(xi, . . . , xq) and std(x„ + i, . . . , x n ) both denote the empty permutation e Sq. 
Thus, for example, 



This map of graded vector spaces from K[5oo] to K[5oo] <8> K[5oo] is a graded co- 
product, because it restricts to a map from K[S n ] to ^2 p+q=n ^[S p ] ® K[5 g ]. 

The graded vector space K[S' 00 ], endowed with the product »s and coproduct As, 
are a graded Hopf algebra (K[S oo], »s, As), meaning that the product satisfies the 
axioms of a graded algebra, the coproduct satisfies the axioms of a graded coalgebra 
(which are dual to the axioms of an algebra), and the product and coproduct satisfy 
a certain compatibility condition. This is the Malvenuto-Reutenauer Hopf algebra 
of permutations, which we refer to as MR. 




n 




A s (563241) = <g> 563241 + 1 <g> 53241 + 12 ® 3241 

+ 231 <g> 231 + 3421 <g> 21 + 45213 <g> 1 + 563241 ® 0. 
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4321 




3241 2431 3412 4213 4132 




3214 2341 3142 2413 4123 1432 




2314 3124 2143 1342 1423 




1234 



Figure 1. The weak order on £4 



2.2. Automorphisms and antiautomorphisms. An important theme running 
through the study of MR is the weak order. The weak order on permutations is the 
partial order whose cover relations are described as follows: Given 
S n , one moves up by a cover by swapping two adjacent entries Xj and x i+ \ with 
Xj < Xj + i. (That is, the higher permutation in the covering pair agrees with x, 
except that the values Xj and Xj + i , which occur in x in numerical order, are swapped 
so as to occur in the higher permutation out of numerical order.) The weak order 
on S4 is illustrated in Figure 1. 

The importance of the weak order to the Malvenuto-Reutenauer Hopf algebra 
begins with the observation [23, Theorem 4.1] that the product x»gy is the sum over 
all permutations in a certain interval in the weak order on S p+q . The connection to 
the weak order also leads us to consider how automorphisms and antiautomorphisms 
of the weak order interact with the product and coproduct. 

Borrowing from the terminology of Coxeter groups, for each n, let Wq be the 
permutation n(n — 1) • • • 21, which is the unique maximal element of the weak 
order. The map x ^ w ■ x is an antiautomorphism of the weak order. (Here, "•" 
is the usual composition product of permutations, and we follow the convention of 
composing permutations as functions, from right to left.) We write rv for this map, 
because the effect of the map is to reverse the values of the one-line notation for 
x, by replacing each value Xj by n + 1 — Xj. For example, rv(625431) = 654321 • 
625431 = 152346. Extending the map rv by linearity to a map on K[5oo]. we 
obtain an algebra antiautomorphism (i.e. an antiautomorphism of the product) 
and a coalgebra automorphism (i.e. an automorphism of the coproduct) of MR. 
The following examples are representative: 

rv(12) • s rv(312) = 21 » s 132 

= 21354 + 23154 + 23514 + 23541 

+ 32154 + 32514 + 32541 + 35214 + 35241 + 35421 
= rv(45312 + 43512 + 43152 + 43125 

+ 34512 + 34152 + 34125 + 31452 + 31425 + 31245) 
= rv(312. s 12). 
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A s (rv(214536)) = A s (563241) 



= <g> 563241 + 1 <g> 53241 + 12 <g> 3241 



+ 231 <g> 231 + 3421 <g> 21 + 45213 <g> 1 + 563241 <g> 
= (rv® rv)(0 <g> 214536 + 1 <g> 13425 + 21 <g> 2314 



+ 213 <g> 213 + 2134 <g> 12 + 21453 <g> 1 + 214536 <g> 0) 
= (rv®rv)(A s (214536)). 



The map x ^ x ■ w is also an antiautomorphism of the weak order. We write 
rp for this map, because the map reverses positions of the entries of the one-line 
notation for x. For example, rp(624531) = 624531-654321 = 135426. Extending by 
linearity, we obtain an algebra automorphism and a coalgebra antiautomorphism 
of MR, as illustrated by the following representative examples: 



where tw is the linear map that takes a ® b to b <g> a. The maps rv and rp com- 
mute, and the composition rv o rp is an antiautomorphism of MR (as a graded Hopf 
algebra, i.e. both as an algebra and as a coalgebra). 

2.3. Self-duality. The weak order defined above is more specifically known as the 
right weak order. There is also a left weak order on permutations. The right and 
left weak orders are isomorphic via the involution x i->- x^ 1 . For the purposes of 
this paper, we take this isomorphism as the definition of the left weak order. We 
also use the term "weak order" as an abbreviation for "right weak order." The 
Hopf algebra MR is self-dual. More specifically, the map inv : x h-» x^ 1 extends 
linearly to an isomorphism [24] between MR and its graded dual MR*. We now 
explain what this means. 

The graded dual of a graded vector space V = ©„> V n is the graded vector 
space V* = (B„>q V*, where each V* is the dual vector space to V n . If each V n 
is finite, then we can identify V* with V n , and thus identify V* with V. If V 
furthermore has the structure of a graded Hopf algebra, then V* is also a graded 
Hopf algebra, the graded dual of V. We define the graded dual Hopf algebra in the 
context of K[Soo], using a permutation x as a shorthand both for the basis element 
indexed by x and for the dual basis element indexed by x. 



rp(12) m s rp(231) 



21 » s 132 

21354 + 23154 + 23514 + 23541 

+ 32154 + 32514 + 32541 + 35214 + 35241 + 35421 
rp(45312 + 45132 + 41532 + 14532 

+ 45123 + 41523 + 14523 + 41253 + 14253 + 12453) 
rp(12. s 231). 



A s (rp(142365)) 



A s (563241) 

® 563241 + 1 O 53241 + 12 <g> 3241 

+ 231 <g) 231 + 3421 <g> 21 + 45213 <g> 1 + 563241 ® 
(rp rp) (0 ® 142365 + 1 <g> 14235 + 21 <g> 1423 

+ 132 O 132 + 1243 ® 12 + 31254 <g> 1 + 142365 <g> 0) 
two (rp®rp) (A 5 (142365)), 
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Dual to the product •§ (a map from K[5oo] ® K[5oo] to K[5oo]) is a coproduct 
• g (a map from K[5oo] to K[5' 00 ] £g> K[5' 00 ]) denned as follows: Given permutations 
z G S n , x e S p and y e 5 g , the coefficient of x ® y in "5(2;) equals the coefficient of 
z in x mg y. Similarly, there is a product A* s dual to the coproduct A5 such that 
the coefficient of z in xA* s y equals the coefficient of x <g> y in Ag(z). 

It is immediate that the map inv extends linearly to an automorphism of graded 
vector spaces of K[5oo]. To see that this is an isomorphism between the graded 
Hopf algebra MR= (K[5oo], »s, A s ) and its graded dual MR* = QK^], A* s , •%), 
we must check that (inv £g> inv)oAs = (•* J )oinv and that inv o(» s ) = A*;o(inv ® inv). 
Both conditions amount to checking that the coefficient of x <£> y in As(z) equals 
the coefficient of in x^ 1 »g y^ 1 for all x, y, and z. This is shown, for example 
in [3, Theorem 9.1]. 

Remark 2.1. The original definition of [24, 25] constructs what we are calling the 
dual Hopf algebra MR* . We prefer the definition given here because it allows us to 
work with the right weak order, rather than translating the results of [28, 29, 30] 
into results about the weak order. The difference in the two definitions is minor, 
in the sense that one can switch definitions by replacing all permutations by their 
inverses. 

3. Some Hopf subalgebras and quotients of MR 

Many important combinatorial Hopf algebras are Hopf subalgebras or quotients 
of MR. One of the most important is the Hopf algebra of Noncommutative symmet- 
ric functions [14], which can be defined on a graded vector space with basis elements 
at degree n indexed by subsets of [n — 1] . Another important Hopf subalgebra of 
MR is the Loday-Ronco Hopf algebra, defined on a graded vector space with basis 
elements indexed by planar binary trees. This is isomorphic [4, 13, 20, 27] to the 
noncommutative Connes-Kreimer Hopf algebra of [13, Section 5]. 

In this section, we describe these Hopf subalgebras, with particular emphasis on 
the Loday-Ronco Hopf algebra. Our description is fairly detailed, for two reasons: 
First, the detailed description gives context to the discussion of the Hopf algebra 
tBax of twisted Baxter permutations/Hopf algebra dRec of diagonal rectangulations 
discussed in Sections 4 and 5. Second, many constructions and definitions developed 
in this section are used directly, in the later sections, in analyzing tBax and dRec. 

None of the results in this section are new, but they are presented in a way 
particularly suited to our purposes. For different accounts of the same material, 
see, for example, [4, 22, 23]. 

3.1. The Hopf algebras LR and LR'. A planar binary tree is a rooted tree (a 
tree with a distinguished vertex called the root) in which every vertex either has 
no children or has two children, one designated as a right child and one as a left 
child. Vertices with no children are called leaves. For the moment, we draw the 
trees with their leaves lined up horizontally and the root above them, with edges 
drawn so that, for each leaf, the path from that leaf to the root is monotone up. 
We call planar binary trees drawn in this manner top trees, for reasons that are 
apparent in Section 6.1. Let PBT^ be the set of top planar binary trees with n + 1 
leaves. For example, the five trees of PBT| are shown in Figure 2. 

To realize the Loday-Ronco Hopf algebra of planar binary trees as a Hopf subal- 
gebra of MR, we define a map p t : S n — > PBT^. Versions of this map have appeared 
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/9x /\ X\ A\ st\ 

Figure 2. The five top planar binary trees in PBT3. 
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Figure 3. Steps in the construction of p t (467198352) 



in many papers, including [7, 8, 22, 23, 30, 33]. Let x = X\x% ■ ■ ■ x n <E S n . Write 
n + 1 points on a horizontal line and label the spaces between points 1,2, ... ,n, 
from left to right. The map pt constructs the tree starting at the leaves and uses 
each entry of x to combine leaves into subtrees, as follows. The two leaves adjacent 
to the label x\ are first joined to a vertex between and above them. The number 
X\ — 1 is then thought of as labeling the space between the new vertex and the 
leaf to its left. Similarly, x\ + 1 labels the space between the new vertex and the 
leaf to its right. The process continues until the last entry of x is read, causing 
the addition of a final new vertex, the root. Figure 3 illustrates the steps in the 
construction of pt(x) when x = 467198352. 

For any (top) planar binary tree T G PBT^, let c(T) be the following sum in 
K[5oo]: 

c(T)= J2 x - 

Pt(x)=T 

The elements of the form c(T) constitute a graded basis for a graded Hopf subalge- 
bra of MR, called the Loday-Ronco Hopf algebra LR. The Hopf algebra LR can be 
viewed as a Hopf algebra on the graded vector space K[PBT^J = 0„> o K[PBT*J. 

Remark 3.1. In [22, 23], the dual definition of MR is taken. (See Remark 2.1.) The 
construction of LR in [22] proceeds via a map ip which sends a permutation x to 
the reflection, through a horizontal line, of the tree p t (iuv(x)). 
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FIGURE 4. Steps in the construction of p b (467198352) 



There is an anti-isomorphic copy LR' of LR that can be realized in MR in a 
similar way. To construct this anti-isomorphic copy, we now draw bottom planar 
binary trees. That is, we draw the trees with the leaves lined up horizontally and 
the root below them, with edges drawn so that, for each leaf, the path from that 
leaf to the root is monotone down. The notation PBT^ stands for the set of bottom 
planar binary trees with n+1 leaves. Now LR' is constructed in a similar manner to 
the construction of LR, using a different map pb from permutations to trees. Once 
again, write n+1 points on a horizontal line and label the spaces between points 
1,2, ... ,n from left to right. We may as well, right from the start, draw a straight 
line from the root to the leftmost leaf, and a straight line from the root to the 
rightmost leaf. A planar binary tree has a uniquely defined left subtree and right 
subtree. The tree Pb(x) is constructed so that x± labels the unique space between 
the two subtrees. Then x% labels the space between two sub-subtrees, etc. Step by 
step construction of pb{x) in the case x = 467198352 £ Sg is illustrated in Figure 4. 

Define for any bottom planar binary tree T £ PBT^ an element c'(T) € K[5oo]: 

c'(T)= Yl x - 

x£S n 
Pb(x)=T 

The elements of the form c'(T) form the graded basis of a graded Hopf subal- 
gebra LR' of MR. We view LR' as a Hopf algebra on the graded vector space 
K[PBT^J = n>o K[PBT^J. The anti-isomorphism between LR and LR' is de- 
scribed in Section 3.2. 

3.2. A pattern-avoidance description. In [30, Section 4], a map r\ is described 
from permutations in S n to triangulations of a convex (n+2)-gon Q. (When reading 
[30, Section 4] solely for the definition of 77, one may safely skip any paragraph 
containing the phrase "fiber polytope" and take the few remaining paragraphs as 
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Figure 5. r,-(467198352) and its dual tree p b (467198352) 



a description of Q and 77.) The vertices of Q are labeled through n + 1, with 
certain conditions on the configuration of the vertices, and the map 77 depends on 
the labeling/configuration. We are first interested in a specific choice of Q (and 
below, in another specific choice). Namely, take the vertices and n + 1 to form 
a horizontal edge of Q, with all of the remaining vertices above that edge, with 
labels increasing left to right. The combinatorics of this choice is summarized, in 
the notation of [30, Section 4], by putting an upper-bar on each index in [n]. 

Given a permutation x € S n , one readily verifies from the definition of r\ that 
the dual planar binary tree to rj(x), when properly "straightened out," is pb{x). 
For an example (and as an illustration of the notion of a dual tree), compare 
Figure 5 to Figure 4. Since a triangulation of Q is determined uniquely by its dual 
tree, we can rephrase (and specialize) results on 77 from [30, Section 5] to obtain a 
characterization of the fibers of pt- 

This characterization uses the fact that the weak order on S n is a lattice. A 
congruence on a lattice is an equivalence relation respecting meet and join in the 
same sense that a congruence of rings respects addition and multiplication. The 
proposition below follows from [30, Theorem 5.1]. 

Proposition 3.2. The fibers of pi, are the congruence classes of a congruence on 
the weak order on S n . 

By a general fact on congruences of finite lattices (see, for example, [29, Sec- 
tion 2]), Proposition 3.2 implies that each fiber of pb is an interval in the weak order. 
In particular, each fiber has a unique minimal representative and a unique maximal 
representative. These representatives can be described by pattern avoidance. For 
1 < k < n, let x — x\Xi ■ ■ ■ xj~ be a permutation in Sk and let y = y\yi ■ ■ ■ y n be 
a permutation in S n . We say y contains the pattern x\— x%— • ■ --Xk if there exist 
indices 1 < i\ < 12 < ■ ■ ■ < ik < n such that < yi t if and only if Xj < xi for all 
j, I G [k]. Otherwise we say y avoids the pattern x\—xi— • • • -x^. Here, we are using 
the generalized pattern notation of [5] . In the more common notation, the dashes in 
X\— x<2.- ■ ■ ■ - Xk are omitted and one speaks of y containing or avoiding X1X2 • ■ ■ Xk- 
The reasons for this more involved notation are explained later, in Section 4.2. See 
[36] for more details on pattern avoidance. The proposition below follows from [30, 
Proposition 5.7]. 

Proposition 3.3. 

(1) A permutation is minimal in its pb-fiber if and only if it avoids 2-3-1. 
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(2) A permutation is maximal in its pb- fiber if and only if it avoids 2-1-3. 

A complete description of the fibers is as follows: Suppose x < y in the weak 
order. Then there exist a pair of entries a and c in y with a immediately following 
c in y and a < c, such that x agrees with y everywhere except that a and c are 
swapped. We say that x is obtained from y by a (231 — > 213) -move if the entries a 
and c are the "1" and "3" in some 231-pattern, or in other words, if there exists an 
entry b with a < b < c such that b occurs somewhere left of ca in y. We also say 
that y is obtained from i by a (213 — > 231) -move. The proposition below follows 
from [30, Proposition 5.3]. 

Proposition 3.4. Let x < y in the weak order. Then pb(x) = Pb{y) if and only if 
x is obtained from y by a (231 — > 2H)-move. 

Let 6231 be the congruence whose classes are the fibers of pi,. Let 7r| 31 stand 
for the map which sends an element x € S n to the minimal element of its 6231- 
class. Let 7r| 31 be the map sending x to the maximal element of its 9 2 3i-class. 
Propositions 3.3 and 3.4, along with the fact that 623i-classes are intervals, imply 
the following recursive characterizations of the maps 7r| 31 and 7r| 31 . 

Proposition 3.5. Let y <G S n . If some permutation x € S n can be obtained from y 
by a (231 -> 213)-move, then 7r| 31 (y) = 7r| 31 (x). Otherwise, 7r^ 31 (y) = y. 

Proposition 3.6. Let x € S n . If some permutation y £ S n can be obtained from x 
by a (213 — > 231)-move, then 7r| 31 (x) = 7r| 31 (y). Otherwise, tt1 31 (x) = x. 

The concatenation of Proposition 3.5 with the previous two propositions implies 
in particular that y admits a (231 — > 213)-move if and only if it contains the pattern 
2-3-1. This is easily checked directly. (See, for example, [30, Lemma 5.5].) A similar 
statement holds for the pattern 2-1-3. 

These considerations allow us to think of LR' as the Hopf algebra of (2-3-1)- 
avoiding permutations, using the map 7r^ 31 in place of pb- The quotient of the weak 
order modulo the congruence 023i is isomorphic [8, 30] to the well-known Tamari 
lattice first defined in [32] . By another general fact about finite lattices (found, for 
example, in [29, Section 2]), this quotient is isomorphic to the subposet induced 
by the set of minimal congruence class representatives. In other words, it is the 
restriction of the weak order to (2-3-l)-avoiding permutations. This restriction is 
also a sublattice of the weak order [8]. The Tamari lattice plays a role for LR' 
analogous to the role the weak order plays for MR. 

All of this development can be adapted to the map p t , which corresponds to 
a different instance of the map n. In this case, we define a polygon similarly to 
the polygon Q described above, with the vertices 1, ...,n occurring below the 
horizontal edge — n + 1. This corresponds, in the notation of [30, Section 4], to 
putting a lower-bar on each index in [n]. Using results of [30, Section 5] in a similar 
way, we obtain analogous results for p t . 

Proposition 3.7. The fibers of p t are the congruence classes of a congruence on 
the weak order on S n . 

Proposition 3.8. 

(1) A permutation is minimal in its p t -fiber if and only if it avoids 3-1-2. 

(2) A permutation is maximal in its p t -fiber if and only if it avoids 1-3-2. 
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Modifying the definitions above in the obvious ways to obtain the definitions of 
(312 — » 132)-moves, (132 — »■ 312)-moves, 6312, 7r^ 12 , and 7r| 12 , we also have the 
following results. 

Proposition 3.9. Let x < y in the weak order. Then pt(x) = Pt(y) "if and only if 
x is obtained from y by a (312 — > 132)-move. 

Proposition 3.10. Let y € S n . If some permutation x <G S n can be obtained from 
y by a (312 -> 132) -move, then it^ 12 (y) = tt^ 12 (x). Otherwise, ir^ 12 (y) = y. 

Proposition 3.11. Let x e S n . If some permutation y S S n can be obtained from 
x by a (132 — > 312)-move, then tt1 12 (x) — ^312(2/)- Otherwise, 7r| 12 (x) = x. 

In particular, LR is the Hopf algebra of (3-l-2)-avoiding permutations. The 
quotient of the weak order modulo O312 is the restriction of the weak order to (3- 
l-2)-avoiding permutations. This quotient is also isomorphic to the Tamari lattice, 
is also a sublattice of the weak order, and plays a role for LR analogous to the role 
the weak order plays for MR. 

The characterization of LR and LR' in terms of pattern-avoidance now makes it 
easy to prove the earlier assertion that they are anti-isomorphic. Recall from Sec- 
tion 2.2 that the map rv o rp is an antiautomorphism of MR. One easily verifies that 
rvorp maps a (231 — > 213)-move to a (312 — » 132)-move. Thus by Propositions 3.4 
and 3.9 (which completely characterize the fibers of pb and p t because congruence 
classes are intervals), we see that rvorp maps the fibers of pi, to the fibers of p t . 
Thus the involution rv o rp restricts to an anti-isomorphism from LR' to LR. 

3.3. The Hopf algebra NSym. Just as LR and LR' were defined using maps 
from permutations to planar binary trees, the Hopf algebra of subsets is also defined 
using a map. In this case, we map to pairs (U, n) such that n > and U is a set 
of positive integers strictly less than n. In particular, U is empty when n = or 
n = 1. Given a permutation x £ S n , define a(x) = (U(x),n), where U(x) is the set 
of positive integers a < n such that a + 1 occurs somewhere before a in X1X2 • • • x n . 

The map a restricts to a lattice homomorphism from the weak order on S n to 
the lattice of subsets of [n — 1] . To see why, recall a convenient characterization of 
the meet and join operations on the weak order. A permutation x e S n is uniquely 
determined by its (left) inversion set: the set of pairs (a, b) such that a > b and a 
occurs somewhere before b in x\x 2 ■ ■ ■ x n . Interpreting this set of pairs as a relation, 
the inversion set of x V y is the transitive closure of the union of the inversion sets 
of x and y. The set U(x) is the intersection of the inversion set of x with the set 
{(a + 1, a) : a e [n — 1]}. It is thus easily seen that U(xVy) = U(x)U U(y). The 
meet in the weak order has a similar characterization involving the transitive closure 
of the union of sets of "non-inversions," and we easily conclude that U(x A y) = 
U(x)r\U(y). 

Since a is a lattice homomorphism, the fibers of a are a lattice congruence, which 
we represent by the symbol O su b- This congruence is the join of 6231 and O312, as 
we now explain. The congruence lattice of a lattice L is the set of all congruences, 
partially ordered as an induced subposet of the lattice of set partitions of L. In 
fact, this subposet is a sublattice of the lattice of set partitions, and furthermore, 
is a distributive lattice. (See, for example, [16, Theorem II. 3. 11].) The following 
fact is immediate from the definition of a: 
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Proposition 3.12. Let x < y in the weak order, so that x is obtained from y by 
swapping a pair of entries in adjacent positions. Then cr(x) — a(y) if and only if 
the two entries differ in value by two or more. 

Proposition 3.12 can be restated as follows: If x < y in the weak order, then 
cr(x) — cr(y) if and only x is obtained from y by either a (231 — > 213)-move or a 
(312 — > 132)-move. Comparing with Propositions 3.4 and 3.9, and keeping in mind 
that su b-classes are intervals, we conclude that O su b is the join of O231 and O312 
in the lattice of partitions of S n , and therefore also in the congruence lattice of the 
weak order on S n . 

The map a shares the remarkable Hopf- algebraic properties of pb and pt- The 
sums over the fibers of a are a graded basis for a graded Hopf subalgebra of MR. 
The Hopf subalgebra, denoted here by NSym, is better known as the Hopf algebra 
of noncommutative symmetric functions [14]. The lattice of subsets play the anal- 
ogous role for NSym as the weak order and Tamari lattice play for MR and LR. 
Alternately, the basis elements of NSym can be indexed by permutations avoiding 
both 2-3-1 and 3-1-2, or equivalently, permutations such that each entry is at most 
one greater than the entry to its right. The considerations at the end of Section 3.2 
show that the map rv o rp is an antiautomorphism of NSym. The same argument 
that shows that 9 su b = 6231 V 3 i 2 also shows that NSym is the intersection of LR 
and LR', as Hopf subalgebras of MR. 

3.4. Some quotients of MR. Passing to graded duals turns Hopf subalgebra 
relationships into quotient relationships. Thus the self-duality of MR means that, 
for each Hopf subalgebra of MR, there is a corresponding quotient Hopf algebra 
of MR (or, in other words, a homomorphic image of MR). In particular, the duals 
LR*, (LR')* and NSym* arc quotients of MR=MR*. 

Consider the map LR°->MR< — >MR* -^>LR*, where the left arrow is the inclusion 
defined in Section 3.1, the right arrow is its dual surjection, and the middle arrow 
is the isomorphism inv. This is a homomorphism of graded Hopf algebras, because 
each of its three steps is a homomorphism. It is also an invertible map of graded 
vector spaces [17, 18], and thus it is an isomorphism. Therefore LR is self-dual. 
The Hopf algebra NSym is not self-dual; its dual NSym* is better known as the 
Hopf algebra of quasi-symmetric functions. 

4. The Hopf algebra tBax of twisted Baxter permutations 

The realizations, given in Section 3, of the Hopf subalgebras LR, LR' and NSym 
of MR all ultimately rest on natural combinatorial maps. Furthermore, these 
maps are lattice homomorphisms from the weak order to natural lattice structures. 
In [29] , a unification and generalization of this construction is given. We begin this 
section with a brief, undetailed summary of the approach and results of [29] . We 
then describe in more detail a particular new Hopf algebra which arose from the 
generalization. 

4.1. "H-Families. For each n > 0, let 0„ be a lattice congruence on the weak order 
on S n . Let (n±) n be the map taking each x e S n to the minimal representative 
of its G„-class. Let Z® — (^i) n (S n ) be the set of minimal elements of 0„-classes. 
(Thus, by a general theorem mentioned earlier, the weak order restricted to 
is isomorphic to the lattice quotient of S n modulo 0„.) Define a graded vector 
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space K[Z®] = ©„> K[Z®]. For each fixed choice of a sequence of congruences, 
define a map c e : K[Z® ] — > K[5oo] by sending each element of Z® to the sum (with 
coefficient 1) over its 0„-class. Define another map r e : K[5oo] — > K[Z^] which, 
for each x € S n , fixes a; if a; € Z® and otherwise sends x € S n to zero. The letters 
c and r suggest "class" and "representative." The map c is an injection, while r is 
a surjection, and the composition roc is the identity on K[Z®], while cor is the 
identity on c e (K[Z°]). 

The graded vector spaces LR, LR', and NSym can all be realized as 
for choices of 0„ described in Section 3. Moreover, in each case, the elements of 
the form c @ (z) form a graded basis of a graded Hopf subalgebra of MR. The main 
result of the last half of [29] is a characterization of families {8„} n>0 of congruences 
having this same property. A family {B„} n>0 is said to be an "H-family if it satisfies 
certain conditions given in [29]. If {9„} n>0 is an "H-family, then the elements c e (z) 
form a graded basis of a graded Hopf subalgebra of MR. In this case, K[Z® ] has a 
unique Hopf algebra structure such that c e is an injective homomorphism of Hopf 
algebras. The product x » z y is r e (x •§ y)- In other words, x *z y is obtained by 
summing over all shifted shuffles of x and y, but ignoring those shuffles that are 
not in some Z®. The coproduct is A z = (r e ®r 9 )oAjoc e . In other words, the 
coproduct has the following formula, which corrects an unfortunate typographical 
error 1 of Equation (2) of [29, Section 8]. 

n 

A z (x)= ^2std e (y 1 ,...,y i )®std e (y l+1 ,...,y n ). 

yes n i=0 

Here ir^y is short for (7rj.) n (y), and std e is the composition r o std. In other words, 
std e standardizes the sequence, and sends the standardization to zero if it is not 
in some Z @ . 

For any "H-family {0ri}„> O i the elements of Z® are described by a pattern- 
avoidance condition given explicitly in [29, Theorem 9.3]. Special cases of the 
condition have been seen in Sections 3.2 and 3.3. Another special case is described 
in Section 4.2. Each map (7rj.)„ is a lattice homomorphism from the weak order on 
all of S n to the weak order restricted to Z® , the set of permutations satisfying the 
pattern-avoidance condition. 

4.2. Twisted Baxter permutations. We now define twisted Baxter permuta- 
tions in the generalized pattern notation of [5]. An occurrence of the pattern 
3-41-2 in a permutation x = x\X2 ■ ■ ■ x n is a subsequence cdab of X1X2 • ■ ■ x n with 
a < b < c < d such that d occurs immediately previous to a in X1X2 ■ ■ ■ x n . This 
differs from an occurrence of the pattern 3-4-1-2 in x, in the sense defined in Sec- 
tion 3.2, which is any subsequence cdab of X1X2 • ■ ■ x n with a < b < c < d, with 
no requirement that d and a be adjacent. Thus for example, the pattern 3-4-1-2 
occurs in 45312 e S5 but the pattern 3-41-2 does not. Similarly, one can define 
2-41-3-patterns, 3-14-2-patterns, and 2-14-3-patterns. 

Twisted Baxter permutations are permutations that avoid both 3-41-2 and 2-41-3. 
They were introduced in [29] as the basis of a new Hopf algebra arising from the 



Equation (2) of [29, Section 8] , the symbols xi , . . . , x p and x p +i , . . . , x n should be replaced 
by j/i, . . . ,y p and y p+1 , . ..,y„. 
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construction described in Section 4.1. They were conjectured in [29] to be in bijcc- 
tion with the (reduced) Baxter permutations of [10], and this conjecture was soon 
after proved by West [35] using generating trees. In Section 8.1, we provide a dif- 
ferent proof. A permutation is a Baxter permutation if and only if it avoids both 
3-14-2 and 2-41-3. Notice that the definition of the twisted Baxter permutations 
is a very slight modification of the definition of Baxter permutations. Let tBax„ 
be the set of twisted Baxter permutations in S n and let Bax n be the set of Baxter 
permutations in S n . 

The patterns 3-41-2, 2-41-3, 3-14-2, and 2-14-3 have the following special prop- 
erty: 

Proposition 4.1. If a permutation x contains the pattern 3-41-2, then x contains 
an instance of the pattern 3-41-2 such that the "3" and the "2" differ in value by 
exactly 1. The same is true of the patterns 2-41-3, 3-14-2, and 2-14-3. 

The assertion of the proposition is that if a; = X\ - ■ -x n contains 3-41-2 then 
x\---x n contains a subsequence (6 + l)dab with a<b<b+l<d and d adjacent 
to a in x\ ■ ■ ■ x n . 

Proof. Suppose cdab is a subsequence of x\ ■ ■ ■ x n with a < b < c < d, such that d is 
adjacent to a. Furthermore, among all such subsequences, choose cdab to minimize 
c — b. Suppose for the sake of contradiction that c ^ b + 1 . If b + 1 occurs to the left 
of d, then (b+l)dab is an instance of the pattern 3-41-2, contradicting our choice of 
cdab. Otherwise 6+1 occurs to the right of a, and cda(b + 1) is an instance of the 
pattern 3-41-2, again contradicting our choice of cdab. This contradiction in both 
cases proves that c = b + 1. We have proved the statement for the pattern 3-41-2. 
The proof for the other patterns is essentially identical. □ 

Proposition 4.1 has the following corollary, which explains why the definition 
above appears to define the inverses of the Baxter permutations defined in [10]. 

Corollary 4.2. A permutation x is a Baxter permutation if and only if the inverse 
permutation x^ 1 is a Baxter permutation. 

Proof. If x contains the pattern 3-14-2, then by Proposition 4.1, x contains a subse- 
quence (b+ l)adb with a < 6 < 6+ 1 < d and a adjacent to d. Write x^ 1 = y\ ■ ■ ■ y n . 
Then the subsequence y a ybyb+iUd of t/i ■ ■ ■ y n is an instance of the pattern 2-41-3. If 
x contains 2-41-3, then by Proposition 4.1, x contains a subsequence bda(b+l) with 
a < b < 6+1 < d and d adjacent to a. In this case, the subsequence y a ybyb+iyd is an 
instance of the pattern 3-14-2. In either case, x~ l is not a Baxter permutation. □ 

A further property of twisted Baxter permutations and Baxter permutations 
is helpful. Given a permutation x = x\---x n € S n and a subset V C [n], the 
restriction of x to the values V is the subsequence of X\ ■ ■ ■ x n consisting of entries 
in V. Note that this is a restriction of x, thought of as a sequence, not a restriction 
of the map x : [n] — > [n]. 

Proposition 4.3. If x £ tBax„ and < p < n, then the restriction of x to the 
values l,...,p is in tBax p , and the standardization of the restriction of x to the 
values p + 1, . . . ,n is in tBax„_ p . 

Proof. To prove the first assertion, suppose to the contrary that the restriction of 
x to l,...,p contains a subsequence a6cd which is an instance either of the pattern 
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3-41-2 or the pattern 2-41-3. Since x avoids both 3-41-2 and 2-41-3, there are entries 
of x between b and c. These entries are all > p. If e is the rightmost such entry, 
then aecd is either a 3-41-2-pattern or a 2-41-3-pattern in x, and this contradiction 
proves the assertion. The second assertion is proved similarly, except that we choose 
e to be the leftmost entry between b and c and conclude that abed is a forbidden 
pattern in x. □ 



An essentially identical argument proves the following proposition. 

Proposition 4.4. If x <E Bax„ and < p < n, then the restriction of x to the 
values 1, . . . ,p is in B&x p , and the standardization of the restriction of x to the 
values p + 1, . . . ,n is in Bax„_ p . 

4.3. The Hopf algebra tBax. The definition of twisted Baxter permutations is a 
special case of the general pattern-avoidance condition of [29, Theorem 9.3]. Thus, 
the twisted Baxter permutations are the minimal congruence class representatives of 
an "H-family of congruences. We refer to the congruences in this family collectively 
as 0tB- The associate downward projection map, taking x G S n to the minimal 
element of its 0tB-class, is denoted by 7r| B , and the analogous upward projection 
map is irl B . Let y <G S n . We say that x is obtained from y by a (3412 — > 3142)- 
move if there exists an instance cdab of the pattern 3-41-2 in y and if x is obtained 
from y by exchanging d and a. In this case, in particular x < y. We also say, in 
this case, that y is obtained from x by a (3142 — >■ 3412) -move. The notions of 
(2413 — > 2143)-mcwes and (2143 — > 2413)-mcwes arc defined analogously. 

As a special case of results of [29, Section 9], the congruences t B have the 
following properties, the first of which is a repetition for emphasis. 

Proposition 4.5. 

(1) A permutation is the minimal element in its Q t B~ class if and only if it is a 
twisted Baxter permutation (i.e., if and only if it avoids 3-41-2 and 2-41-3). 

(2) A permutation is the maximal element in its Q t B-class if and only if it 
avoids 3-14-2 and 2-14-3. 

(3) Suppose x<y in the weak order. Then x = y modulo O t e if and only if x is 
obtained from y by either a (3412 — > 3142) -move or a (2413 — > 2143) -move. 

(4) let y G S n . If some permutation x G S n can be obtained from y by a 
(3412 -> 3142) -move or a (2413 -> 2143) -move, then 7r| B (y) = tt| b (.t). 
Otherwise, 7r' B (y) = y. 

(5) Let x e S n . If some permutation y e S n can be obtained from x by a 
(3142 3412) -move or a (2143 -> 2413) -move, then 7r t T B (x) = 7r t T B (y). 
Otherwise, 7t^ b (.t) = x. 

The twisted Baxter permutations constitute a graded basis for a Hopf subalgebra 
tBax of MR. As a special case of the description in Section 4.1, the product and 
coproduct in tBax have the following descriptions. The product x » t B y of two 
twisted Baxter permutations x and y is the sum of all twisted Baxter permutations 
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that are obtained as shifted shuffles of x and y. Thus, for example, 

21 » tB 21 = 2143 + 3444 + 2431 + 4213 + 4231 + 4321 

= 2143 + 2431 + 4213 + 4231 + 4321 

21 « t B 132 = 21354 + 23154 + 34544 + 23541 

+ 32154 + 32544 + 32541 + 35244 + 45344 + 35421 

= 21354 + 23154 + 23541 + 32154 + 32541 + 35421 

For a twisted Baxter permutation z, the coproduct A t B(z) is obtained in two 
steps. First sum Ag(w) over all permutations w in the 9 t B-class of z. Then delete 
all terms which contain permutations that are not twisted Baxter. For example, 
let z = 532641 e tBax 6 . By Proposition 4.5, we calculate the t B-class of z to be 
{532641,536241,563241}. 

A s (563241) = ® 563241 + 1 <g> 53241 + 12 <g> 3241 

+ 231 <g> 231 4 3421 ® 21 + 45213 <g> 1 4 563241 ® 

A s (536241) = ® 536241 + 1 <g> 35241 + 21 ® 4231 

+ 213 <g> 231 + 3241 <g> 21 + 42513 <g> 1 4 536241 <g> 

A s (532641) = ® 532641 + 1 <g> 32541 + 21 ® 2431 

+ 321 <g> 321 + 3214 <g> 21 + 42153 <g> 1 4 532641 <g> 0. 

Thus, 

A tB (532641) = 00 563241 + 1 <g> 53241 + 12 ® 3241 + 231 <g> 231 4 3421 ® 21 

+ 45213 ® 1 + 563241 ® + ® 536211 + 1 © 35211 

+ 21 ® 4231 + 213 ® 231 + 3241 ® 21 + 42513® 1 

+ 536241 ® + ® 532641 + 1 ® 32541 + 21 ® 2431 

+ 321 ® 321 + 3214 ® 21 + 42153 ® 1 + 532641 ® 

= ® 532641 + 1 ® (53241 + 32541) + 12 ® 3241 

+ 21 ® (4231 + 2431) + (213 + 231) ® 231 4 321 ® 321 

+ (3421 + 3241 + 3214) ® 21 + 42153 ® 1 + 532641 ® 

Notice that all terms in A tB (532641) occur with coefficient 1. This turns out to 
be true in general. 

Proposition 4.6. Let z e tBax„. Then each term occurring with nonzero coeffi- 
cient in A t B(z) occurs with coefficient 1. 

Proof. Suppose v and w are distinct permutations in the 0tB-class of z. It is enough 
to show that, for any p with < p < n, either std(«i, . . . , v p ) ^ std(wi, . . . , w p ) or 
std(w p+ i, . . . , v n ) 7^ std(w p +i, . . . , w n ) or both. 

Fix such a p and let y = v A w, the meet of v and w in the weak order. Since OtB 
is a lattice congruence, y is also in the 6 t B-class of z. Choose i to be the smallest 
positive index such that either vt differs from yi or Wi differs from yi or both. Since 
v and w are distinct, such an i exists and is less than n. We claim that v; t ^ u>i. 

Suppose to the contrary that Vi = Wi and let k be such that yt = u». Then k > i, 
because v and y agree in positions left of position i. For every j with i < j < k, the 
element yj precedes the entry Vi in y but follows in v. Since y < v in the weak or- 
der, and since the weak order is the containment order on inversion sets, we conclude 
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that yj < Vi = y k - Let y' be the permutation y\ ■ ■ ■ yi-iykVi • • • Vh-iVk+i •••y n , ob- 
tained by removing y k from y and re-inserting it between and y^. Passing from 
y to y' creates some inversions, but destroys no inversions, so y' > y. The inversions 
created in y' are inversions of v, so y' < v. By the same argument, y' < w. This 
contradicts our choice of y to be the meet of v and w, thus proving the claim. 

If p < i then std(v p+ i, . . . , v n ) ^ std(io p +i, . . . , w n ) because v and w are distinct, 
so assume p > i. By the claim, we take, without loss of generality, Vi > Wi. 
We complete the proof by showing that there exists I with 1 < I < i such that 
Wi < vi < Vi. Since wi = vi, this implies that std(«i, . . . , v p ) ^ std(wi, . . . , w p ). 

Since (vi, Wi) is not an inversion in w, it is also not an inversion in y. Consider a 
saturated chain in the weak order from y to v. Since (v i} Wi) is not an inversion in 
y but is an inversion of v, as we follow the chain from y up to v, we eventually come 
to an edge x < x' such that x and x' differ only in the order of the adjacent entries 
Vi and Wi. Since OtB-classes are intervals, the entire chain is contained in the same 
0tB-class, so Proposition 4.5.3 says in particular that there exists an entry a to the 
left of Wi and Vi in x and x' with Wi < a < Vi. Choose a to be the largest entry to 
the left of Wi and with Wi < a < v^. If a is not vi for some 1 < I < i, then as we 
proceed further up the chain from x' to v, we eventually come to an edge u < u' 
corresponding to swapping adjacent entries a and Vi. But then Proposition 4.5.3 
says that there exists an entry a' to the left of a and i>, in u and v! with a < a' < 
Since u is weakly higher up the chain than x' and since (v i7 a') is not an inversion 
in u, we conclude that (vi, a') is not an inversion in x and x'. Thus a' is to the left 
of Vi in x and x' , contradicting our choice of a to be maximal. This contradiction 
shows that a = vi for some 1 < I < i, thus completing the proof. □ 

The Hopf algebra tBax is closely related to LR and LR'. Recall from Section 3.3 
that the congruence 6 su b, which defines the Hopf algebra NSym, is the join of 
the congruences 8231 and 6312 in the lattice of partitions of S n , and thus in the 
lattice of congruences of the weak order on S n . The congruence 6 t e, which defines 
the Hopf subalgebra tBax, is just the opposite. The following proposition is [29, 
Proposition 10.2]. 

Proposition 4.7. The congruence O t B is the meet 0/6231 and 6312 in the lattice 
of congruences of the weak order on S n ( or equivalently in the lattice of partitions 
ofS n ). 

That is, two permutations are congruent modulo OtB if and only if they are 
congruent modulo 6231 and modulo 8312- 

Remark 4.8. Proposition 4.7 is a weaker statement than the statement that tBax 
is the smallest Hopf subalgebra of MR containing LR and LR'. However, we have 
no evidence against the stronger statement. 

Remark 4.9. Several automorphisms and antiautomorphisms of tBax are inherited 
from MR. The algebra automorphism/coalgebra antiautomorphism rp of MR takes 
(3412 -> 3142)-moves to (2143 -> 2413)-movcs and takes (2413 2143)-moves 
to (3142 — > 3412)-moves, and thus, by Proposition 4.5.3, restricts to an algebra 
automorphism/coalgebra antiautomorphism of tBax. The algebra antiautomor- 
phism/coalgcbra automorphism rv of MR has the same effect on (3412 — > 3142)- 
moves and (2413 — > 2143)-moves, and thus restricts to an algebra antiautomor- 
phism/coalgebra automorphism of tBax. Their composition rvorp is a Hopf alge- 
bra antiautomorphism of tBax. 
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Remark 4.10. Recall from Section 3.4 that the Hopf algebra LR is self-dual, and 
that its self-duality can be proved by verifying that the Hopf algebra homomor- 
phisms LRM-MR-f^-MR* -^>LR* is an invertible map of graded vector spaces. We 
remark that the analogous argument fails when applied to tBax. That is, the map 
tBax^MR< — >MR* -^>tBax* is not invertible. For example, one can verify that the 
matrix of the restriction of this map to tBax4 has two identical rows. Of course, the 
failure of this argument does not rule out the existence of some other isomorphism 
between tBax and tBax*. 

5. The Hopf algebra of diagonal rectangulations 

As described above, the Hopf algebra of twisted Baxter permutations has, like 
LR, LR', and NSym, a realization in terms of pattern avoiding permutations. The 
product and coproduct are calculated by computing products and coproducts in 
MR, and then deleting terms which don't satisfy the avoidance conditions. However, 
LR, LR', and NSym admit intrinsic descriptions of their products and coproducts 
in terms of the fundamental combinatorial objects (planar binary trees and subsets) 
indexing their bases, without first embedding into a larger Hopf algebra [23, 14]. In 
this section, we supply a similar combinatorial realization of the Hopf algebra tBax. 
Specifically, we define diagonal rectangulations and define a product and coproduct 
on the graded vector space spanned by diagonal rectangulations. The product and 
coproduct on diagonal rectangulations define a Hopf algebra, which we call dRcc. 
The proof that dRec is a Hopf algebra appears in Section 6, where it is shown that 
dRec is isomorphic to the Hopf algebra tBax. 

5.1. Diagonal rectangulations. The general notion of a rectangulation of a rec- 
tangle arises in integrated circuit design. (See references in [1, Section 1].) We are 
interested in a particular class of rectangulations of a square which we call diago- 
nal rectangulations. Ackcrman, Barequet, and Pinter [1, Lemma 5.2] showed that 
diagonal rectangulations are counted by the Baxter number, by showing that they 
satisfy a recurrence from [10]. (A proof using nonintersecting lattice paths is given 
in [12].) Thus diagonal rectangulations emerge as a candidate to index the basis of 
a Hopf algebra isomorphic to tBax. 

Let S be a square with sides parallel to the coordinate axes. Consider a subdi- 
vision of S into finitely many rectangles. Necessarily, all the sides of the rectangles 
are also parallel to the coordinate axes. We define the walls of the decomposition 
to be the maximal line segments contained in the union of the boundaries of the 
rectangles, excepting the four edges of S, which are not considered to be walls of 
the decomposition. The interior of the line segment connecting the top-left corner 
of S to the bottom-right corner is called the diagonal of S. Let A be a set of 
n — I distinct points on the diagonal of S. A diagonal rectangulation of (S, X) is a 
subdivision of S into rectangles such that no two walls intersect in their interiors, 
such that every wall of the decomposition contains a point of X, and such that 
every point of X lies on a wall. 

Proposition 5.1. A diagonal rectangulation of(S,X) with \X\ = n—1 has exactly 
n — 1 walls and has exactly n rectangles. 

Proof. Let R be a diagonal rectangulation. Because the points of X are distinct 
points on the diagonal, every wall defining R contains exactly one point in X. 
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Figure 6. An illustration for the proof of Proposition 5.1. 



Suppose two walls contain the same point x in X. Since the walls have disjoint 
interiors, x is an endpoint of both. By the definition of the walls of R, the two 
walls are not contained in the same line, and in particular, there are only two walls 
containing x. But now, considering R locally about x, we see that R is not a 
decomposition into rectangles, and this contradiction shows that every point in X 
is on exactly one wall of R. Thus R has n — 1 walls. 

We conclude the proof by arguing that any decomposition of S into rectangles 
with n — 1 interior-disjoint walls has exactly n rectangles. Indeed, let L be any 
wall in such a decomposition. There are two walls (or edges of S) whose interiors 
contain the endpoint of L, and some walls L%, . . . , whose endpoints are contained 
in the interior of L. The situation is illustrated in Figure 6. a, where L is the red 
(or gray) wall. The number of rectangles incident to the interior of L is k + 2. 
Removing L and extending the walls L\, . , . ,L^, as illustrated in Figure 6.b, we 
obtain a decomposition of S into rectangles with n — 2 walls. By induction, there 
are n — 1 rectangles in the new decomposition. The number of rectangles in the 
new decomposition intersecting the interior of L is k + 1, and thus the original 
decomposition has n rectangles. □ 



The precise size of S and the exact choice of points in X is irrelevant to the com- 
binatorics of diagonal rectangulations. Indeed, given a rectangulation R of (S, X), 
any dilation applied to S can also be applied to X and R to give a well-defined 
rectangulation of the dilated square. Furthermore, given any choice X' of n — 1 
points on the diagonal of S, there is a natural choice of a rectangulation R' of 
(5, X') corresponding to R. (This is most easily understood by considering shifting 
one point of X at a time, without moving it past any other point of X.) These 
considerations define an equivalence relation on the set of all diagonal rectangula- 
tions of pairs (S, X), where S and X vary. The fundamental combinatorial object 
is an equivalence class of such rectangulations. We call such an equivalence class a 
diagonal rectangulation of size n. For convenience, we blur the distinction between 
diagonal rectangulations of size n (i.e. equivalence classes) and equivalence-class 
representatives. For definiteness, we often consider the integral representative of 
the equivalence class. This is the representative such that S = [0,n] x [0,n] and 
X is the set of integer points on the diagonal. In particular, we often specify a 
diagonal rectangulation by a representative picture. Figure 7 shows (the integral 
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Figure 7. A rectangulation of size 20 



representative of) a diagonal rectangulation of size 20. The diagonal is shown in 
gray. 

An alternate characterization of diagonal rectangulations is useful. 

Proposition 5.2. A decomposition of a square into rectangles is a diagonal rect- 
angulation of size n if and only if it has n rectangles and each rectangle 's interior 
intersects the diagonal. 

Proof. Suppose R is a diagonal rectangulation of size n. Then Proposition 5.1 says 
that R has n rectangles. If one of these rectangles U is contained completely below 
the diagonal, then consider the two walls of R containing the top-right corner of U. 
One of these walls ends at the corner, and in particular that wall contains no point 
of X. This contradiction, and the analogous contradiction for rectangles contained 
above the diagonal, complete one direction of the proof. 

Conversely, suppose that R is a decomposition of S into n rectangles, with each 
rectangle's interior intersecting the diagonal. Moving along the diagonal of S, we 
visit each rectangle exactly once, and there are exactly n — 1 points where we 
cross from one rectangle to another. Let X be the set of these n — 1 points. It 
is immediate that every point in X is contained in a wall of R. To complete the 
proof, we need to show that every wall of R contains a point in X. Equivalently, 
we need to show that every wall of R crosses the diagonal. But if some wall of R 
does not cross the diagonal, then its closest point to the diagonal is the corner of 
some rectangle whose interior docs not intersect the diagonal. □ 

Remark 5.3. Diagonal rectangulations arc in bijection with the Mosaic fioorplans 
defined in [21] in connection with a proposed floorplanning algorithm for VLSI 
circuit design. Consider the class of (not necessarily diagonal) rectangulations with 
exactly n — 1 walls, such that the interiors of walls are disjoint. Mosaic fioorplans 
of size n are equivalence classes of such rectangulations under a certain equivalence 
relation which is coarser than topological equivalence. In [37] , mosaic fioorplans are 
shown to be counted by the Baxter number. As a special case of [1, Theorem 4], each 
mosaic floorplan has exactly one representative that is a diagonal rectangulation. 

5.2. Product and coproduct. Let dRec n stand for the set of rectangulations of 
size n. The set dReco has a single element, the rectangulation of a x square 
having no rectangles. This empty rectangulation is represented by the symbol 0. 
The set dReci also has a single element, the "decomposition" of a square into a 
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Figure 8. A product calculation in the Hopf algebra dRec 



single square. The Hopf algebra dRec is the graded vector space KfdReCoo] = 
©n>o K[dRec n ], with the product and coproduct that we now describe. 

Given a set of interior-disjoint, axis-parallel line segments in S, a diagonal rect- 
angulation R is a completion of the set of line segments if every line segment in the 
set is contained in a wall of R. A principal subsquare of the square S is a square S' 
contained in S such that the diagonal of S' is contained in the diagonal of S. Given 
a diagonal rectangulation R of S and a principal subsquare S' of S, the restriction 
R! of R to S' is the following decomposition of S 1 into rectangles: Each rectangle of 
R! is an intersection UDS' such that U is a rectangle of R and UOS' has nonempty 
interior. Each rectangle of R' crosses the diagonal of S' , and thus R' is a diagonal 
rectangulation by Proposition 5.2. 

Let p be an integer with < p < n. Consider the integral representative of a 
diagonal rectangulation R, as defined in Section 5.1. (Thus S = [0, n] x [0, n] and X 
is the set of integer points on the diagonal.) Define the p th top-left restriction of R 
to be the diagonal rectangulation TL p (R) of size p obtained as the restriction of R 
to the principal subsquare [0,p] x [n — p, n\. For an integer q with < q < n, define 
the q th bottom-right restriction of R to be the diagonal rectangulation BR g (i?) of 
size q obtained as the restriction of R to the principal subsquare [n — q, n] x [0, q\. 

The product in dRec, written «dR or • , is described as follows: If Ri £ dRec p and 
R% £ dRcc g , where p+q = n, then i?i»dR^?2 is the sum over all rectangulations R € 
dRec„ such that TL p (i?) = i?i and BR 9 (i?) = R 2 . For the purpose of computations, 
the product -Ri'dR-Ra can a ls° be described as follows. Represent R\ and R 2 by their 
integral representations on the respective squares [0,p] x [0,p] and [0,q] x [0,q], and 
represent rectangulations appearing in R\ »dR i?2 by their integral representations 
on [0,n] x [0, n]. Take the p — 1 walls of i?i and translate them upwards q units, 
and take the q — 1 walls of R 2 and translate them p units to the right. The product 
is the sum of those diagonal rectangulations of size n which are completions of the 
union of the two sets of line segments, such that the n — 1 walls of the completion 
are obtained by extending (if necessary) the n — 2 segments already present and 
adding a new wall containing the diagonal point (p, n — p). We give two examples 
in Figures 8 and 9, marking, for the sake of clarity, the point (p, n — p) in each 
diagonal rectangulation in the product. 

The coproduct on rectangulations is written AdR or simply A. Let R be a 
diagonal rectangulation of size n and consider any path 7 from the top-left corner 
of S to the bottom-right corner of S, traveling only along edges of rectangles, and 
traveling only downwards and to the right. The path 7 divides the rectangles of 
R into two groups: There are p rectangles below/left of the path and q rectangles 
above/right of the path, with p + q = n. If p > 0, then by an easy induction on p, 
we conclude that there are exactly p—1 walls of R that intersect the interior of the 
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FIGURE 9. A product calculation in the Hopf algebra dRec 



region in S below/left of 7. Similarly, if q > 0, then there are exactly q — 1 walls of 
R that intersect the interior of the region in S above/right of 7. 

We associate to the path 7 an element A 1 ® B 1 of dRec^dRec. If p = then the 
element A 7 is 0. Otherwise, A 1 is obtained as follows: First, delete from R all parts 
of walls that lie on or above/right of 7. Exactly p — 1 line segments remain. The 
element A 1 is the sum over all completions of the collection of p— 1 line segments to 
a diagonal rectangulation of size p. The walls of any such completion are obtained 
by extending, if necessary, the line segments already present, without adding any 
additional walls. Similarly, B-y = if q = 0, and otherwise B~ i is obtained by 
deleting from R all part of walls that lie on or below/left of 7, and then summing 
over all completions of the resulting collection of q — 1 line segments to a diagonal 
rectangulation of size q. The coproduct of R is the sum, over all paths 7, of A^®B^. 

Consider, for example, the coproduct of 



Figure 10 shows each path 7 and the associated A 1 and B 1 . The figure uses the 
abbreviation "c." to mean "the sum over all completions of appropriate size." 

The example in Figure 10 illustrates an important property of the coproduct 
AdR, which we now prove in general. 

Proposition 5.4. Let R be a diagonal rectangulation. Then each term occurring 
with nonzero coefficient in AdR(i?) occurs with coefficient 1. 

Proof. By definition, for each fixed path 7 from the top-left corner of R to the 
bottom-right corner of R, every nonzero coefficient in the sums A 7 and B 1 is 1. 
Suppose 71 and 72 are distinct paths from the top-left corner of R to the bottom- 
right corner of R. The term ® R occurs in AdR(i?) with coefficient 1, so we need 
not consider the case where 71 or 72 passes through the bottom-left corner of S. Let 
R\ be a decomposition of S obtained as a completion of the segments below 71 and 
let i?2 be a decomposition of S obtained as a completion of the segments below 72. 
To prove the proposition, it is enough to show that Ri and R2 are distinct diagonal 
rectangulations. 

Starting from the top-left corner of R, consider the first point x at which 71 
and 72 diverge. Without loss of generality, 71 continues downward from x and 72 
continues to the right from x. At some point y directly below x, the path 71 turns 
to the right. There are four possible local configurations at y: The point y can be 
in the interior of the right edge of a rectangle of R, in the interior of the left edge 
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FIGURE 10. A coproduct calculation in the Hopf algebra dRec 
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of 5, in the interior of the top edge of a rectangle of R, or in the interior of the 
bottom edge of S. 

Suppose y is in the interior of the right edge of a rectangle U of R. The de- 
composition Ri has a rectangle U\ containing U, and the decomposition R 2 has a 
rectangle U 2 containing U. Since Ri and R 2 are identical below and to the left of 
y, if the decompositions R\ and R 2 are the same diagonal rectangulation, then the 
rectangles U\ and U 2 correspond. For each horizontal wall W of R having its left 
endpoint contained in the right edge of U, strictly lower than the point y, there 
is a horizontal wall of i?i having the same left endpoint (contained in the right 
edge of Ui), and furthermore, these are the only horizontal walls of R\ having left 
endpoints contained in the right edge oiU\. For each of these horizontal walls W 
of R with left endpoint in the right edge of U, strictly lower than y, there is also 
a horizontal wall of R 2 with the same left endpoint (contained in the right edge of 
U 2 ). But there is at least one more horizontal wall whose left endpoint is contained 
in the right edge of U 2 , namely a horizontal wall of R 2 whose left endpoint is y. 
We conclude that R\ and R 2 are not the same diagonal rectangulation. 

If y is in the interior of the left edge of S, then there is no rectangle of R to the 
left of y, but by considering walls whose left endpoint is in the left edge of S, we 
reach the same conclusion. 

Now suppose y is in the interior of the top edge of a rectangle U of R, let U\ be 
the rectangle of R\ containing U, and let U 2 be the rectangle of R 2 containing U. 
Every vertical wall of R\ whose bottom endpoint is contained in the top edge of U\ 
corresponds to a vertical wall of R, lying to the left of y, whose bottom endpoint 
is contained in the top edge of U. For each of these vertical walls of R there is a 
vertical wall of R 2 whose bottom endpoint is contained in the top edge of U 2 , but 
there is at least one additional vertical wall of R 2 with bottom endpoint in the top 
edge of U 2 : the wall whose bottom endpoint is y. Thus R\ and R 2 are not the same 
diagonal rectangulation. 

If y is in the interior of the bottom edge of S, then there is no rectangle of R 
below y, but arguing similarly we reach the same conclusion. □ 

6. The isomorphism from tBax to dRec 

In this section, we give a bijection between twisted Baxter permutations and 
diagonal rectangulations and show that the product and coproduct on diagonal 
rectangulations, defined in Section 5.2, coincide, via the bijection, with the product 
and coproduct on tBax. In particular, dRec is a Hopf algebra. We then consider 
some natural automorphisms and antiautomorphisms of the Hopf algebra dRec. 

6.1. Bijection. The bijection from twisted Baxter permutations to diagonal rect- 
angulations is the restriction of a surjection p from S n to dRec„. Given x G S n , 
the diagonal rectangulation p(x) is obtained by "concatenating" the bottom planar 
binary tree pb{x) with the top planar binary tree pt{x) as we now make precise: 
Construct both trees with their leaves evenly spaced at intervals of y/2, with all lines 
drawn straight and all angles right. Rotate both trees clockwise by an eighth-turn, 
place the root of Pb{x) at the origin and place the root of pt{x) at the point (n, n). 
Inspection of the definitions of p\, and pt leads immediately to the conclusion that 
the union of the edges of the two rotated and translated trees defines a decomposi- 
tion of the square into n rectangles, each of whose interiors intersects the diagonal. 
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By Proposition 5.2, this is a diagonal rectangulation of size n. Inspection of these 
definitions also leads immediately to the following direct description of the map p. 

Let x G S n . Start with n — 1 diagonal points in the square S and number the 
spaces between them 1,2, ... ,n from top- left to bottom-right. For convenience in 
the description of p, we also include the top-left and bottom-right corners of S as 
"diagonal points." We read x\---x n from left to right and draw a rectangle for 
each entry. After i — 1 steps in the construction, let Tj_i be the union of the left 
and bottom edges of S with the union of the rectangles drawn in the first i — 1 
steps. Once the construction is described, it is apparent by induction that each 
Tj_i is left- and bottom-justified. We draw an additional rectangle whose top- left 
and bottom-right corners are described as follows: 

Consider the label Xi on the diagonal. If the diagonal point p immediately 
above/left of the label Xj is not in Tj_i, then the top- left corner of the new rectangle 
is the rightmost point of Tj_i that is directly left of p. If p is in Tj_i (necessarily on 
the boundary of Tj_i, then the top- left corner of the new rectangle is the highest 
point of Tj_i directly above p. Similarly, if the diagonal point p' immediately 
below/right of the label X{ is not in T,_i, then the bottom-right corner of the new 
rectangle is the highest point of Tj_i that is directly below p. If p' is in T^_\ then 
the bottom-right corner of the new rectangle is the rightmost point of Tj_i that is 
directly to the right of p' . Figure 11 illustrates the steps in the construction of p{x) 
when x = 467198352. In each step, the new rectangle is shown in red (the darkest 
gray when not viewed in color), and the set Tj_i is shaded in medium gray. The 
part of S not yet covered by rectangles is shaded in light gray. 

Our next goal is to prove the following theorem: 

Theorem 6.1. The map p restricts to a bijection between twisted Baxter permuta- 
tions in S n and diagonal rectangulations of size n. 

Theorem 6.1 follows from two propositions. 

Proposition 6.2. The map p : S n — > dRec„ is surjective. 

Proof. Let R be a diagonal rectangulation. We construct a permutation x £ S n 
with p{x) = R. By Proposition 5.2, R is a decomposition of the square into n 
rectangles, each of whose interiors intersects the diagonal. Label the rectangles of 
R as 1, 2, . . . , n from top- left to bottom-right along the diagonal. 

For i € [n], assume that the entries x\x 2 • • • Xj-i have been determined, and that 
steps 1 through i — 1 in the description of p, applied to x x x 2 ■ ■ ■ give exactly 
the rectangles of R labeled by {xix 2 ■ ■ ■ a^-i}- We try to choose a rectangle whose 
label can be taken as X{. Let T be the set Tj_i in the description of the i^ step of 
p. That is, T is the union of the left and bottom edges of S with the rectangles of 
R labeled by {xix 2 ■ ■ ■ 

The analogous set Tj must in particular be left- and bottom-justified. Thus the 
rectangle we choose in step i must have both its entire left edge and its entire 
bottom edge contained in T. We claim that this condition on the rectangle is also 
sufficient. In other words, we claim that, as long as we take Xi to label a rectangle 
U whose left and bottom edges are in T, the i th step of p reproduces U. To see 
this, we need only verify that the top-left and bottom-right corners of U are as 
described in the definition of p. 

The boundary of U intersects the diagonal in exactly two points. Let p be the 
top-left of the two, and let p' be the bottom-right of the two. If p £ T then 
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Figure 11. Steps in the construction of p(467198352) 



necessarily p is in the left edge of U. If the top corner of U is somewhere below the 
highest point of T directly above p, then the top-left corner of U is the bottom-left 
corner of some rectangle U' of R, but then the interior of U' does not intersect the 
diagonal, which would be a contradiction. If p T, then since the left edge of U 
is contained in T, the point p is on the top edge of U. Since the left edge of U is 
in T, the top-left corner of U is in T, and is necessarily the rightmost point of T 
that is directly left of p. In either case, we see that the top-left corner of U is the 
point described in the direct description of p. The symmetric argument proves the 
analogous fact about the bottom-right corner of U, and the claim is proved. 

In light of the claim, it remains only to argue that there exists some rectangle 
U of R which is not contained in T but whose bottom and left edges are contained 
in T, The boundary of T includes a polygonal path from the top-left corner of S 
to the bottom-right corner of S, always moving directly right or directly down. At 
each corner where the path turns from moving down to moving right, there is a 
rectangle of R which is not contained in T. Number these rectangles U±, . . . , Uj. 
from top-left to bottom-right. We show that there is a j € [k] such that the left and 
bottom edges of Uj are contained in T. The following three simple observations 
are illustrated in Figure 12. First, the left edge of U\ is contained in T. Second, 
for every j £ [k — 1], if the bottom edge of Uj is not contained in T, then the left 
edge of Uj + i is contained in T. Third, the bottom edge of is contained in T. 
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Figure 12. An illustration for the proof of Proposition 6.2 

Thus, considering the rectangles in order U\, . . . , Uk, we eventually find a rectangle 
Uj whose left and bottom edges are contained in T. □ 

Proposition 6.3. The fibers of the map p : S n —¥ dRec„ are the congruence classes 

o/e tB . 

Proof. Proposition 3.2 established that the fibers of pb are the congruence classes of 
a congruence ©231- Similarly, by Proposition 3.7, the fibers of p t form a congruence 
©312- Two permutations x,y £ S n have p{x) = p(y) if and only if both pb(x) = 
Pb{y) and pt(x) = Pt(y)- Thus the fibers of p constitute the meet, in the partition 
lattice, of 6231 and 8312. Proposition 4.7 identifies this meet as 6 t B- □ 

Now, each 6 t B-class has a unique minimal representative, and the set of these 
minimal representatives is the set of twisted Baxter permutations by Proposi- 
tion 4.5.1. This completes the proof of Theorem 6.1. 

The proof of Theorem 6.1 leads easily to an explicit description of the inverse 
of the restriction of p. Let R be a diagonal rectangulation. The proof of Propo- 
sition 6.2 constructs a permutation x, one entry at a time, such that p(x) = R. 
At each step, there were choices. Let t(R) be the permutation obtained by al- 
ways choosing the smallest possible entry at each step. In other words, choose the 
top/leftmost allowable rectangle from the list U±, . . . , Uk- 

Proposition 6.4. For any diagonal rectangulation R, the permutation t(R) is a 
twisted Baxter permutation. Thus, the map t is the inverse of the restriction of p 
to twisted Baxter permutations. 

It is a simple and instructive exercise to suppose that the permutation t(R) 
contains the pattern 3-41-2 or the pattern 2-41-3 and to obtain a contradiction 
(i.e. to show that, at some step, we failed to choose the top/leftmost rectangle). 
However, this proposition is also an easy consequence of Propositions 4.5.1 and 6.3. 

Proof. Let x — X\ ■ ■ ■ x n be any permutation covered by r(i?) = t\ ■ ■ ■ t n in the 
weak order, and let + l be the indices where x and t(R) differ. In particular, 
X\ ■ ■ ■ Xi-i =ti--- ti-i and Xi < U. But U is, by definition, the minimal entry of any 
permutation starting with t\ ■ ■ ■ and mapping to R under p. Thus p(x) 7^ R. 
By Proposition 6.3, the fibers of p are the 6 t B-class, and are in particular intervals. 
We conclude that t(R) is minimal in its 6 t B-class so that t(R) is a twisted Baxter 
permutation by Proposition 4.5.1. □ 
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Remark 6.5. Reflection in the diagonal permutes the set of diagonal rectangula- 
tions. The map p seems not to respect this symmetry, because it builds a diagonal 
rectangulation starting from the bottom-left corner. However, we might just as 
well have built diagonal rectangulations from the top-right corner. Indeed, if pi is 
the map taking x G S n to the diagonal reflection of p(x), one easily verifies that 
p' = p o rp, where rp is the "reverse-positions" map defined in Section 2.2. Thus 
the original map p has an alternate description, reading x from right to left, and 
drawing a top-right-justified rectangle for each entry. 

Remark 6.6. As mentioned in Section 3.1, various versions of the maps from per- 
mutations to trees have appeared in various sources. The idea of mapping a permu- 
tation to a pair of "twin" binary trees to get pairs counted by the Baxter number 
is due to [11], where the pairs are also counted by a non-intersecting lattice path 
argument. The idea of putting the two trees together to form a rectangulation 
appears to be due to [1]. A map essentially equivalent to p is used in [12] in a 
bijection between pairs of twin binary trees and Baxter permutations, via diagonal 
rectangulations. More is said about the map from [12] in Remark 8.5. A map 
related to p also appears in [2, Section 3]. 

Remark 6.7. Each diagonal rectangulation R of size n corresponds to a certain 
partial order on the set [n] . Specifically, first define a partial order on the rectangles 
of R as follows: Put U\ < U2 if the bottom or left edge of U2 intersects the top or 
right edge of U\. The proof of Proposition 6.2 serves to verify that this relation is 
acyclic, so we obtain a partial order on the rectangles of R by taking the reflexive 
and transitive closure. Since the rectangles of R are labeled by [n], this is a partial 
order on [n] which we call the adjacency poset of R. The proof of Proposition 6.2 
also shows that the fiber p^ 1 (R) is the set of all linear extensions of the adjacency 
poset of R. It would be interesting to give an intrinsic characterization of the 
posets that arise as adjacency posets of diagonal rectangulations. In particular, the 
Hopf algebra dRec (or the isomorphic Hopf algebra tBax) could then be given an 
alternate combinatorial realization on the graded vector space with graded basis 
indexed by adjacency posets. 

6.2. Isomorphism. In this section, we prove the following theorem: 

Theorem 6.8. The triple dRec = (K[dRcCoo], »dR, Ajr) is a graded Hopf algebra. 
The map p is an isomorphism of Hopf algebras from tBax to dRec. 

Comparison of the examples in Sections 4.3 and 5.2 provides an illustration of 
the theorem. We need the following lemma. 

Lemma 6.9. Let x G S p and y G S q and let z be a term in x »s y. Then p(x) = 
TL p (p(z)) and p(y) = BR q (p(z)). 

Proof. Let n = p + q. It is immediate from the description of p that the rectangles 
of TL p (p(z)) are the intersections of [0,p] x [q,n] with rectangles constructed from 
entries Zj = i of z with i G {l,...,p}. Furthermore, TL p (p(z)) depends only 
on x, (i.e. only on the relative order of the values 1 through p in z). Similarly, 
BR q (p(z)) depends only on y. Since TL p (p(z)) = p(x) and BR 5 (p(z)) = p(y) when 
z\ ■ ■ ■ z n = x\ ■ ■ ■ x p (yi + p) ■ ■ ■ (y q + p), the lemma follows. □ 

Proof of Theorem 6.8. It is enough to show that p is an algebra homomorphism and 
a coalgebra homomorphism. Then Theorem 6.1 implies that p is an isomorphism, 
and thus dRec is a Hopf algebra because tBax is. 
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We first verify that p(x »tB V) = p{x) »dR p{y)- Let x G tBax p and y G tBax g . 
Since both » tB and »dR are sums with all coefficients one, the equation p(x» t B y) = 
p(x) »dR p(y) holds if and only if the set of terms appearing in p(x » t B y) equals the 
set of terms appearing in p(x) »dR p{y)- Lemma 6.9 says that every term p{z) in 
p(x »tB y) is also a term in p(x) » dR p(y). 

On the other hand, let R be a term in p(x) •dRp(y), so that p(x) = TL p (i?) and 
p(y) = BR q (R). Let z — t(R), so that p(z) = R. The restriction x' of z to the 
values 1, . . . ,p is a twisted Baxter permutation by Proposition 4.3, and Lemma 6.9 
says that p{x') = TL p (R). Since p is a bijection from tBax p to dRec p , we conclude 
that x — x', so that x is the restriction of z to the values 1, . . . ,p. Similarly, y is 
the standardization of the restriction of z to the values p + 1, . . . , n. Thus z is a 
term in x » t B y, and so R is a term in p(x » t B 2/)- 

We now show that (p(g>p)(A tB (z)) = A dR (p(z)). Let z G tBax„ and let R = p(z). 
For p + q = n, let x G tBax p and let y G tBax 9 . In light of Propositions 4.6 and 5.4, 
we only need to show that the term x ® y occurs in A t B (z) if and only if the term 
p(x) C§> p{y) occurs in AdR(-R). 

First, suppose that p(x) (g> p(y) is a term in AdR(-R), and let 7 be the path 
such that p(x) is a term in A 1 and p(y) is a term in £> 7 . We show that x (g> y 
is a term in A t B(z) by exhibiting a permutation w such that a; = std(wi • • -w p ), 
y = std(w p+ i ■ ■ ■ w q ), and p(w) = R. Numbering the rectangles of R, as usual, 
from 1 to n according to their intersections with the diagonal, from top-left to 
bottom-right, let V be the set of numbers whose rectangles appear below/left of 7. 
Then there exists a unique permutation x' of V such that std(x') = x and a unique 
permutation y' of [n] \ T such that std(y') = y. We define w to be the permutation 
in S n such that w\ ■ ■ ■ w p = x' and w p+1 ■ ■ ■ w n = y'. 

The permutation x describes a sequence of rectangles in p(x) such that each 
rectangle in the sequence has its left and bottom edges contained in the union of 
the preceding rectangles (or in the left or bottom edge of S). But the rectangles 
below/left of 7 in R are exactly the rectangles obtained as the intersection of a 
rectangle of p(x) with the region below/left of 7. Thus the sequence x' describes a 
sequence of rectangles in R such that each rectangle in the sequence has its left and 
bottom edges contained in the union of the preceding rectangles (or in the left or 
bottom edge of S) . Arguing as in the proof of Proposition 6.2, the first p steps in the 
construction of p{w) agree with R. In other words, p(w) agrees with R below/left 
of 7. We can apply the same argument to the alternate description of p discussed 
in Remark 6.5 to conclude that the first q steps of the alternate description of p(w) 
agree with R, so that p(w) agrees with R above 7. Thus p(w) = R, or in other 
words, w is in the 0tB-class of z. We conclude that x ® y is a term in A t B(-R)- 

Conversely, suppose x ® y is a term in A tB (z). Thus there exists w in the t B- 
class of z such that x = std(wi • • • w p ) and y = std(w p+ i • • • u>„). Let 7 be the path 
in R defined as the upper-right boundary of the union of the rectangles constructed 
in the first p steps of the construction of p(w), together with the left and bottom 
edges of S. As discussed in connection with the definition of AdR, if p > 0, then 
there are exactly p — 1 walls of R that extend below/left of 7. The p — 1 walls 
meet the diagonal in a set X' of p — 1 distinct points. We now show that p(x), 
constructed on the square S with diagonal points X' , agrees with R below/left of 7. 

We argue by induction on p. The base case p = 1 is easy, so suppose p > 1. 
Let 7 be the path separating rectangles constructed in steps 1 through p — 1 in the 
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construction of p(w) from the other rectangles of R. Let X' be the subset of X 
constructed from 7 just as X' was constructed from 7. Let R be the diagonal rect- 
angulation p(std(wi • • ■ w p -i)), as constructed on S, with respect to the diagonal 
points X'. By induction, R agrees with R in the region strictly below/left of 7. 

Consider the alternate description (see Remark 6.5) of p(x) = p(std(wi • • • w p )), 
as constructed on the square S, with respect to the diagonal points X' and the 
alternate description of R = p(std(wi • • • w p -i)), as constructed with respect to the 
diagonal points X'. Each process constructs a diagonal rectangulation by drawing 
a sequence of top- and right -justified rectangles. Comparison of the two processes 
reveals that R agrees with p(x) in the region S \ U, where U is the first top- 
and right -justified rectangle constructed in the alternate description of p(x). We 
conclude that p{x) agrees with R in the region strictly below/left of 7. The rectangle 
between 7 and 7 is the part of U that is below 7, so p{x) agrees with R below 7. 

In light of the alternate description of p, the symmetric proof shows that p(y) 
and R agree above 7. We have showed that p(x) ® p(y) is a term in AdR,(i?). This 
completes the proof of the theorem. □ 

Remark 6.10. Remark 4.9 discusses automorphisms and antiautomorphisms of 
tBax. These exhibit themselves as (anti)automorphisms of dRec given by sym- 
metries of the underlying square. The algebra antiautomorphism/coalgebra auto- 
morphism rv of tBax corresponds to the algebra antiautomorphism/coalgebra auto- 
morphism of dRcc obtained by reflecting each rectangulation along the line segment 
connecting the lower-left corner of S to the upper-right corner. The algebra auto- 
morphism/coalgebra antiautomorphism rp of tBax corresponds to reflecting each 
rectangulation along the diagonal. (See Remark 6.5.) Thus the Hopf-algebra anti- 
automorphism rv o rp corresponds to rotating each rectangulation one half-turn. 

Remark 6.11. The Hopf algebra LR is embedded as a Hopf subalgcbra of dRec. 
Specifically, a bottom tree T maps to the sum of all diagonal rectangulations whose 
bottom half is T. The analogous map embeds LR' as a Hopf subalgebra of dRec. 
These embeddings can be used to give descriptions of the product and coproduct in 
LR and LR' analogous to the definition of the product and coproduct in dRec. It is 
interesting to compare these descriptions to the descriptions, from [4, Section 1.5], 
of the (dual) product and coproduct in LR* . 

7. The lattice and polytope of diagonal rectangulations 

The weak order on S n modulo the congruence t B is a lattice on the set of 0tB _ 
classes. A general fact about finite lattices, mentioned previously in Section 3.2, 
implies that this quotient is isomorphic to the subposet of the weak order induced 
by the set twisted Baxter permutations. Since 0tB-classes are in bijection with 
diagonal rectangulations, this quotient also defines a lattice structure on diagonal 
rectangulations. We use the symbol dRec„ to stand for the set dRec„ endowed 
with this quotient lattice structure. In this section, we describe the lattice of 
diagonal rectangulations, as a partial order, in terms of local moves on rectangles, 
analogous to the diagonal flips on triangulations that define the Tamari lattice. 
These local moves were already considered in [1, Section 2]. We also construct a 
polytope, analogous to the associahedron, whose vertices are indexed by diagonal 
rectangulations. 
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Figure 13. The lattice dRec4 of diagonal rectangulations of size 4 

As an example of the lattice of diagonal rectangulations, consider the case n = 4. 
The weak order on S4 is illustrated in Figure 1. Applying p to each element of £4, 
we obtain all 22 elements of dRec4. The permutations 3412 and 3142 both map to 
the same diagonal rectangulation and the permutations 2413 and 2143 both map to 
the same diagonal rectangulation. The partial order dRcc4 is shown in Figure 13. 

To describe the partial order dRec„ directly in terms of diagonal rectangulations, 
we need to define some further terminology. Let R be a diagonal rectangulation of 
size n. A point is a vertex of R if it is the vertex of some rectangle in R. Vertices 
are shown as white dots in Figure 14. A line segment E is an edge of R if it is 
contained in the edge of some rectangle of R, if its endpoints are vertices of R and 
if it contains no vertex of R in its interior. 

Every vertex v of R, with the exception of the corners of S, is incident to exactly 
three edges, and we say that v is a(n) up, down, right or left vertex according to 
the direction of the unmatched edge (the edge which is not opposite another edge 
incident to v). The vertices in Figure 14 are labeled "u," "d," "r," or "1" for up, 
down, right or left. Every left or down vertex must be above the diagonal and 
every right or up vertex must be below the diagonal. (Otherwise, some rectangle 
incident to the vertex has its interior disjoint from the diagonal, in contradiction to 
Proposition 5.2.) If a vertex v is not a corner of S, then exactly two edges incident 
to v exit v in the direction of the diagonal. One is the unmatched edge; we say 
that the other edge leaving v towards the diagonal is locked by v. In addition, we 
declare that any edge incident to a corner of S is locked by that corner. It is easily 
verified that, with this convention, every edge contained in the boundary of S is 
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Figure 14. An illustration of terminology related to pivots 



locked by some vertex. The locked edges in Figure 14 are overwritten with hash 
marks. 

An edge of R can participate in a pivot if and only if it is not locked. The pivots 
are described as follows. Let E be a non-locked edge of R. 

If E is an unmatched edge of both of its endpoints, then the endpoints of E are 
necessarily either a left vertex and a right vertex or a down vertex and an up vertex. 
Furthermore, E must, in this case, cross the diagonal, necessarily at a point p in 
the set X. There are two rectangles U\ and U2 of R such that E is an edge of U\ 
and an edge of Ui. To pivot the edge E, remove it and insert the unique other edge 
E' containing p that cuts the rectangle U\ U U2 into two rectangles. The result is 
another diagonal rectangulation R' of size n. The endpoints of E are not vertices 
of R' , but they are replaced by two new vertices, the endpoints of E' . This type of 
pivot is called a diagonal pivot, because E and E' share a point on the diagonal. 

Suppose E is a matched edge of one of its endpoints v. Then since E is not 
locked, E is the matched edge of v which leaves v in the direction away from the 
diagonal, and E is the unmatched edge of its other endpoint v' . In particular, the 
two rectangles U\ and U2 of R whose boundaries contain E together form a hexagon 
with five convex right angles and one concave right angle. To pivot E, remove it 
and insert the unique other edge E' that cuts U\ U U2 into two rectangles. Again, 
the pivot produces a diagonal rectangulation R' of size n. The point v is a vertex 
of R' but v' is not. It is replaced by a new vertex, the other endpoint of E' . This 
type of pivot is called a vertex pivot, because E and E' share a vertex of R and 
of R'. 

In both cases, pivoting the edge E produces a new edge E' in R' , such that E' 
is not locked and such that pivoting the edge E' transforms R' back to R. Also, 
in both vertical edge is replaced by a horizontal edge or vice-versa. There 

are eight rectangulations that are obtained from the diagonal rectangulation of 
Figure 14 by pivoting. These are shown in Figures 15 and 16. In each picture, the 
new edge appears in red (or gray). We now state the main result of this section. 
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Figure 15. Diagonal rectangulations covered by the diagonal 
rectangulation of Figure 14 



Figure 16. Diagonal rectangulations which cover the diagonal 
rectangulation of Figure 14 

Theorem 7.1. Two diagonal rectangulations R and R' of size n have R < R' in 
dRcc„ if and only if they are related by a pivot such that the pivoted edge in R is 
vertical. 

Theorem 7.1 can be proved by appealing to Proposition 4.7 and to descriptions of 
covers in the weak order modulo 023i and in the weak order modulo 6312 that can 
be obtained as specializations of [30, Proposition 5.3]. However, to avoid having to 
translate these descriptions from the language of triangulations to the dual language 
of planar binary trees, we argue directly in terms of diagonal rectangulations. 

Proof. Let L be a finite lattice, let 8 be a congruence on L, and let y be the 
minimal element in its 9-class. Then [29, Proposition 2.2] states that a 0-class C 
is covered by the class of y in L/Q if and only if some element of C is covered by y 
in L. Specializing this fact to the weak order and the congruence OtB, we rephrase 
the theorem as the following assertion: For any y 6 tBax„, R' — p(y) and R are 
related by pivots as in the statement of the theorem if and only if R = p(x) for 
some permutation x covered by y in the weak order. 

Let y € tBax„ and suppose x < y in the weak order. Thus x and y differ only 
in positions i and i + 1, and Xi = yi+\ < y-i — Xi+i. Consider the partial diagonal 
rectangulation created from either x or y in the first i — 1 steps of p and how the 
next two steps proceed for x and y. As in the proof of Proposition 6.2, we let T 
be the union of the left and bottom edges of S with the rectangles produced in the 
first i — 1 steps. We let Xj and X i+X be the rectangles produced in steps i and i + 1 
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Figure 17. An illustration for the proof of Theorem 7.1 



of the construction of p(x) and define Yi and Y i+ i analogously for p(y). Since y 
is a twisted Baxter permutation, one of the following three cases applies. In every 
case, it is apparent that, after step i + 1, the constructions of p{x) and p{y) proceed 
identically. 

Case 1: y% = y-i+i + 1. The rectangles Yi and l^+i have a diagonal point p 
in common, and p is in the interior of an edge of each rectangle. Let L be the 
(necessarily horizontal) edge of R' containing p. (Thus L is the intersection of the 
edge of Yi containing p with the edge of Yj+i containing p.) If the endpoint q of 
L that is above/right of the diagonal is a down vertex, then the rectangle whose 
top- left corner is q must be chosen after Yf but before Y i+1 . If the endpoint r 
of L that is below/left of the diagonal is an up vertex, then the rectangle whose 
bottom-right corner is r must be chosen after Yi but before Y i+i . Since Y i+1 is 
chosen immediately after Yi in the construction of p(y), we conclude that q is a left 
vertex, that r is a right vertex, and therefore that Yj U is a rectangle. The 
same argument shows that Xi U X i+i is a rectangle, and it is easy to see that the 
rectangles Xi U X i+1 and Yi U Y i+ \ coincide. Thus p(x) and p{y) are related by a 
diagonal pivot. 

Case 2: yi > y i+ i + 1 and every a with < a < yi occurs to the left of position i 
in yi • • • y n . In this case, all rectangles labeled a with y i+i < a < yi are in T. Thus, 
following the boundary of T from the bottom- left corner of Yi + \ to the bottom- left 
corner of Yi we move to the right to a point that we call p, and then we move 
down, making no other turns. The point p is the top-left corner of Yi, and the 
top-right corner of Yi is the bottom-right corner of Y i+ i. On the other hand, p is 
the bottom-right corner of JQ, and the top-right corner of Xi is the top-left corner 
of X i+ i. Thus p(x) and p(y) are related by a vertex pivot, where the pivot-vertex 
is p. 

Case 3: yi > yi+\ + 1 and every a with yi+\ < a < yi occurs to the right of position 
i + 1 in y\ ■ ■ ■ y n . In this case, no rectangles labeled a with y i+ i < a < yi are in 
T, and the boundary of T moves first down, then right as we traverse it from the 
top- left corner of Y i+1 to the bottom-right corner of Yj,. Similarly to Case 2, we see 
that p(x) and p(y) are related by a vertex pivot. 

In every case, the pivoted edge is vertical in p(x) and horizontal in p{y). 

Conversely, suppose that y € tBax„ and that R' = p{y) and R are related by 
pivots as in the statement of the theorem. Since the pivoted edge in R is vertical, 
the pivoted edge in R' is horizontal, so one of the rectangles involved in the pivot 
is lower than the other in R' . Call the lower rectangle Yi, and let yi be the entry 
of y associated to Yi by p. The other rectangle, Yj is above Y t and so is associated 
to a later entry yj of y with yj < yi . The rectangles Yi and Yj can stand in three 
possible relationships, illustrated in Figure 17. Again let T be the union of the 
left and bottom edges of S with the rectangles produced in the first i—1 steps of 
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constructing p(y). The region T is indicated in gray in Figure 17. A priori, the left 
edge of Yj may not be contained in T, but we argue below that it is. 

Since y — t(R'), in the sense of Proposition 6.4, y i+ i is the smallest value in 
{yi+i, ■ ■ ■ , y n } such that the rectangle Y i+ i labeled yi + i has its left and bottom edges 
contained in the union T U Yi of the left and bottom edges of S with the rectangles 
chosen in steps 1 to i. We claim that j = i + 1. The claim is equivalent to two 
assertions: First, the assertion that the left and bottom edges of Yj are contained in 
TUYf, and second, the assertion that yj is the smallest value in {j/j+i, . . . , y n } such 
that the rectangle Yj has this property. If either assertion fails, then inspection of 
each case illustrated in Figure 17 reveals that some other rectangle, above/left of 
Yi, could have been chosen in step i, contradicting the fact that y = t(R'). 

In light of the claim, we can take x to be the permutation obtained from y by 
swapping yi and j/j+i, so that x < y in the weak order. Inspection of the three cases 
illustrated in Figure 17 (corrected to show that the left edge of Yj is contained in 
T) shows that R = p(x). □ 

We now describe the construction of a polytope whose vertices are indexed by 
diagonal rectangulations and whose edges correspond to pivots. The construction 
involves fans, particularly normal fans of polytopes; we refer the reader to [38, 
Lecture 7] for background. Consider the arrangement A of hyperplanes in R™ given 
by equations x. t = Xj for each i and j with 1 < i < j < n. This is called the braid 
arrangement or the Coxeter arrangement for S n - The complement of UhgaH m ^™ 
consists of nl connected components, whose closures are called regions, and which 
correspond naturally to the elements of S n : For each permutation X\X 2 • • • x n G S n , 
there is a unique region of A whose interior contains the vector (x\,X2, ■ ■ ■ , x n ) <E 
R™, and each region of A contains exactly one such vector. The regions are the 
maximal cones of a complete fan T , which is the normal fan of a polytope called 
the permutohedron. 

Any lattice congruence 6 on the weak order can be interpreted as an equivalence 
relation on regions. Consider the union of the regions in some equivalence class. 
As a special case of [29, Theorem 5.1], each such union is a convex cone, and 
furthermore these cones are the maximal cones of a fan T% which coarsens T . 
Hohlweg and Lange [19, Proposition 4.1] showed that the fans J"e 231 and J"e 312 
are the normal fans of polytopes which we call P 2 3i and P 3 i 2 respectively. Both 
P231 and P312 are realizations of the associahedron, meaning that their vertices are 
indexed by the set of triangulations of an (n + 2)-gon and their edges correspond to 
diagonal flips. Proposition 4.7 says that ls the coarsest fan that refines both 

Pe 23 i an( i -^0312 ■ Thus a well-known fact on Minkowski sums (for example, see 
[38, Proposition 7.12]) implies that J^Q tB is the normal fan of the polytope, which 
we call -PdRco given by the Minkowski sum P231 + p3i2- Now [29, Proposition 3.3] 
and [29, Theorem 5.1] combine to say that the Hasse diagram of dRec„ orients the 
1-skeleton of PdRoc- Thus by Theorem 7.1, the vertices of PdRcc are indexed by 
diagonal rectangulations and the edges of PdRec correspond to pivots. 

Remark 7.2. Since dRec„ is a finite lattice, its Hasse diagram is in particular con- 
nected. In light of Theorem 7.1, we recover the observation, originally made in 
[1, Lemma 2.3], that the edge-pivoting graph on diagonal rectangulations of size 
n is connected. Since this graph is also the 1-skeleton of the (n — l)-dimensional 
polytope PdRec, Balinski's Theorem (for example, see [38, Theorem 3.14]) implies 
the following stronger result: The graph on diagonal rectangulations of size n is 
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(n — l)-connected, meaning that the removal of at most n — 2 vertices leaves a 
connected graph. 

Remark 7.3. Edge-pivoting preserves the number of vertices of a diagonal rectan- 
gulation. We thus easily conclude that every diagonal rectangulation of size n has 
exactly 2n + 2 vertices. 

Remark 7.4. Beyond the results mentioned above, [29, Theorem 5.1] implies addi- 
tional detailed combinatorial information about dRec„. In particular, open intervals 
in dRec„ are either homotopy equivalent to spheres or are contractible, and thus 
Mobius function values are in { — 1,0, 1}. Furthermore, dRec„ is the partial order 
on the maximal cones of J"e tB induced by a certain linear functional. For definitions 
and details, see [29, Sections 2-5]. 

8. Baxter permutations and twisted Baxter permutations 

In Section 4.2, we saw that the definitions of twisted Baxter permutations and 
Baxter permutations are closely related syntactically. Specifically, recall that a 
permutation is a twisted Baxter permutation if and only if it avoids both 3-41- 
2 and 2-41-3, and that a permutation is a Baxter permutation if and only if it 
avoids both 3-14-2 and 2-41-3. While syntactically similar definitions often produce 
quite different mathematical objects, we have already seen that the two definitions 
produce sets of permutations sharing the same cardinality (the Baxter number). In 
this section, we explain the relationship between Baxter permutations and twisted 
Baxter permutations by giving a bijection that is built directly upon the syntactic 
relationship. As has been the case throughout the paper, lattice congruences of 
the weak order play a key role. The bijection from Baxter permutations to twisted 
Baxter permutations allows us to carry both a natural Hopf algebra structure and 
a natural lattice structure over to Baxter permutations. 

8.1. Bijection. To define the bijection between twisted Baxter permutations and 
Baxter permutations, we appeal to yet another special case of the results of [29, 
Section 9]. There is a lattice congruence 63412 on each S n with associated projection 
maps 7r^ 412 and 7rg 412 such that the family of congruences O3412 is an "H-family, and 
thus has the following properties: 

Proposition 8.1. 

(1) A permutation is the minimal element in its <d 3 4i 2 -class if and only if it 
avoids 3-41-2. 

(2) A permutation is the maximal element in its Q^^-class if and only if it 
avoids 3-14-2. 

(3) Suppose x < y in the weak order. Then x = y modulo O3412 if and only if 
x is obtained from y by a (3412 — > 3142)-mowe. 

(4) Let y G S n - If some permutation x € S n can be obtained from y by a 
(3412 ->• 3142) -move, then Trf 412 {y) = Trf il2 (x). Otherwise, ttJ 4 12 (y) = y. 

(5) Let x G S n . If some permutation y G S n can be obtained from x by a 
(3142 — > 3412)-move ; then ^^(x) = 7r| 412 (y). Otherwise, 7r| 412 (x) = x. 

The main result of this section is the following theorem. 

Theorem 8.2. The restriction of nf 412 is a bijection from Baxter permutations to 
twisted Baxter permutations. Its inverse is the restriction of -k\ a12 . 
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To prove the theorem, we prove two lemmas: 

Lemma 8.3. Suppose x is the bottom element of some ©3412 -c/ass and y is the top 
element of the same 63412-ciass, so that 7r| 412 (y) = x and nl 412 (x) = y. If y avoids 
2-41-3 then x avoids 2-41-3. 

Lemma 8.4. Every <d tB -class contains a Baxter permutation. 

Given Lemmas 8.3 and 8.4, Theorem 8.2 follows as we now explain. 

Proof of Theorem 8.2. Assertions 1 and 2 of Proposition 8.1 say in particular that 
a Baxter permutation is the top element of its 3 4i 2 -class and a twisted Baxter 
permutation is the bottom element of its 34 i2-class. Now Lemma 8.3 implies that 
^3412 ma p S Baxter permutations to twisted Baxter permutations. Thus, since 7r 3412 
is a bijection from top elements of 63412-classes to bottom elements of 3 4i 2 -classes, 
the restriction of tt^ 412 to Baxter permutations is a one-to-one map to the set of 
twisted Baxter permutations. If a; is a twisted Baxter permutation, then Lemma 8.4 
says that there is a Baxter permutation y in the t B-class of x. Thus, 7r| 412 (y) is 
a twisted Baxter permutation. Comparing Propositions 4.5 and 8.1, we see that 
OtB-classes are unions of 03412-classes, so tt| 412 (j/) is in the 0tB~class of x. Since 
each twisted Baxter permutation is the unique minimal element of its 0tB-class, 
we have x = TT^ 412 (y). Furthermore, y — Tt\ il2 {x) because y is the top element of 
its 9 3412 -class. □ 

We now prove Lemmas 8.3 and 8.4. 

Proof of Lemma 8.3. We prove the lemma by proving the following assertion: If y 
is not minimal in its 03412-class and y avoids 2-41-3, then there is a strictly lower 
clement y' in the 3 4i 2 -class of y which also avoids 2-41-3. 

Indeed, a y that is not minimal in its 03412-class contains an instance of the 
pattern 3-41-2. That is, there exists a subsequence cdab of y\ ■ ■ ■ y n with a < b < 
c < d and with d immediately preceding a in y x ■ ■ ■ y n . Let e be the rightmost 
element left of d which is less than d. Possibly e coincides with c, and otherwise 
e is to the right of c. Let /1 • • • /, be the sequence of elements between e and d. 
Possibly i = 0, in which case the sequence fi - ■■ f% is empty. Let h be the leftmost 
element right of a which is greater than a. Let gi ■ ■ ■ gj be the sequence of elements 
between a and h. Again, possibly h — b and/or j = 0. Define a permutation y' by 
removing the sequence agi ■ ■ ■ gj from yi ■ ■ ■ y n and replacing it between e and f\ . 
Thus we write 



b ■ ■ ■ Vn 




<d <a >d >a 

Since every entry in the sequence a g\ • • ■ g 3 is < a and every element of the sequence 
/1 • • • fi d is > d, we can obtain y' from y by a sequence of (3412 — > 3142)-moves. 
In each move, c plays the role of "3," an entry from the sequence f\ - ■ ■ fid plays 
the role of "4," an entry from ag\ ■ ■ ■ gj plays the role of "1," and b plays the role 
of "2." Thus by Proposition 8.1, y' is in the 3 4i 2 -class of y. 
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We now show that if y' contains a 2-41-3-pattern, then y also contains a 2-41-3- 
pattern. Suppose that y' contains a subsequence /3<5ct7 that is a 2-41-3-pattern. 
That is, a < (5 < 7 < 6 and S immediately precedes amy'. We break into six easy 
cases, based on the location of amy'. 

Case 1: a is e or some entry to the left of e. Then (3Saj is a 2-41-3-pattern in y 
as well. 

Case 2: a coincides with a. Then 5 coincides with e and 7 < S = e < d. Since 
each / is > d, no / coincides with 7, and furthermore 7 is not d. Thus fldaj is a 
2-41-3-pattern in y. 

Case 3: a coincides with some g. Then 5 either a or some g, and therefore 5 < a. 
In particular, since each / is > d, and since j < 5 < a < d, no / coincides with 7. 
Therefore f35a-f is a 2-41-3-pattcrn in y as well. 

Case 4: a coincides with some / or with d. Note that a does not coincide with fi 
(and if j — 0, then a is not d) because 5 immediately precedes a and S > a. Now 
a > d > a, but each g is < a, so (3 is not some g and is not a. Therefore /36 cry is a 
2-41-3-pattern in y as well. 

Case 5: a is ft. Then 5 is d. Also, a > a, so (3 > a > a, and thus /3 does not 

coincide with any g nor with a. Thus /3da7 is a 2-41-3-pattcrn in y. 

Case 6: a is some entry right of h. Then (35aj is a 2-41-3-pattern in y as well. □ 

Proof of Lemma 8.4- By Proposition 6.3, the 9 t B-classes are the fibers of p. Thus 
it is enough to show that, for every diagonal rectangulation R, there is a Baxter 
permutation x with p(x) — R. As in the proof of Proposition 6.2, we construct the 
permutation one entry at a time. In other words, we totally order the rectangles 
of R so that each rectangle has its left and bottoms sides contained in the union of 
the previous rectangles with the left and bottom sides of S. 

To order the rectangles, we use an algorithm first given in [12]. (See Remark 8.5 
below.) Let U be the rectangle chosen in step i — let p be the top-right corner of 
U, and let Tj_i be the union of the rectangles chosen in steps 1 through i — 1 with 
the left and bottom sides of S. Follow the boundary of Tj_i left from p until it turns 
upward, and let Ua be the rectangle whose bottom-left corner is the point where 
the boundary turns upward. If the top edge of U is contained in the boundary of S, 
then Ua is not defined. Similarly, follow the boundary of Tj_i down from p until 
it turns towards the right side of S, and let Ub be the rectangle whose bottom-left 
corner is the point where the boundary turns. If the right edge of U is contained 
in the boundary of S, then Ub is not defined. If both the top and right edges of U 
are in the boundary of S, then i — l = n and we are done. Otherwise, the top-right 
corner p of U is either a left vertex or a down vertex. If p is a left vertex, then 
choose Ua as the next rectangle in the total order, and if p is a down vertex, then 
choose Ub next. (This rule is consistent with the cases where only Ua or only Ub 
is defined.) 

To complete the proof, we need to establish two assertions: First, we need to 
show that it is possible to follow this algorithm to obtain a permutation x with 
p{x) = R. Specifically, we need to show that, at every step i, the algorithm succeeds 
in selecting a rectangle whose left and bottom sides are contained in Tj_i. Second, 
we need to show that the permutation x is a Baxter permutation. 

Suppose that we have successfully followed the algorithm in steps 1 to i — 1, 
choosing rectangle U is step i — Suppose that the top- right corner p of U is 
a left vertex, so that Ua is chosen in step i. Since p is a left vertex, there is a 
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rectangle V whose bottom-right corner is p. (It is possible that V = Ua-) Let L 
be the vertical line containing p. The key point in the argument is the following 
immediate observation: For every j > i, if we have successfully continued the 
algorithm in steps i through j — 1 without ever choosing rectangle V, then the 
rectangle chosen in step j has its interior completely to the left of L. 

We now show that, if we have successfully followed the algorithm in steps i — 
then the algorithm also succeeds, in step i, in picking a rectangle with its left and 
bottom sides contained in Tj_i. By the symmetry of reflection perpendicular to the 
diagonal, we may as well assume that p is a left vertex, so that the chosen rectangle 
is Ua- Since p is a left vertex, the bottom edge of Ua is contained in Tj_i. Suppose 
that the left edge of Ua is not contained in Tj_i. Let p' be the highest point in the 
intersection of Ua with Tj_i, let U' be the rectangle whose top-right corner is p' 
and let V be the rectangle whose bottom-right corner is p' . In particular, V has 
not been chosen before step i, because p' is the highest point in the intersection 
of Ua with Tj_i. Since U was chosen just before Ua, it was in particular chosen 
after U', but then since p' is a left vertex, the observation of the previous paragraph 
says in particular that U could not have been chosen before V. This contradiction 
shows that the left edge of Ua is contained in T. 

We have shown that the algorithm constructs a permutation x with p{x) = R. 
It remains to show that the permutation a; is a Baxter permutation. Suppose x 
contains a 3-14-2-pattcrn: a subsequence XhXi-iXiXj of x\ ■ ■ ■ x n such that x,_i < 
Xj <Xh < Xi. As before, let U be the rectangle added in step i—1 and let p, Tj_i, 
and Uq be as above. Since the next rectangle chosen is labeled Xj, this rectangle is 
Ub- In particular, moving downwards from p along the boundary of Tj_i. we pass 
to the right of the rectangle U' chosen in step h. But the rectangle U" chosen later, 
in step j, is between U and U' on the diagonal, and since Tj_i is left- and bottom- 
justified we see that the interiors of U" and Tj_i intersect. This contradiction 
implies that x contains no 3-14-2-pattern. If x contains a 2-41-3-pattern, then we 
reach a similar contradiction by an argument that is symmetric with respect to 
reflection perpendicular to the diagonal. □ 

Remark 8.5. The algorithm given in the proof of Lemma 8.4 is essentially bor- 
rowed from the proof of [12, Theorem 5.1], which gives a bijection between Baxter 
permutations and twin binary trees. The ideas used above to prove that the al- 
gorithm works and produces a Baxter permutation are also essentially the same. 
However, the algorithm in [12, Theorem 5.1] outputs the inverse of the permuta- 
tion constructed in the proof of Lemma 8.4. For this reason, for completeness, and 
also because of the differences between Theorem 8.2 and [12, Theorem 5.1], we 
reproduce the argument here. 

8.2. Lattice and Hopf structures on Baxter permutations. In this section, 
use Theorem 8.2 to study the subposet (in fact, lattice) of the weak order induced by 
Baxter permutations, and to put a Hopf algebra structure on Baxter permutations. 

In a general finite lattice, it is well known, and easy to verify, that the downward 
and upward projection maps associated to a congruence are order-preserving. Fur- 
thermore, the quotient modulo a congruence is isomorphic not only to the subposet 
induced by the bottom elements of congruence classes, but also, by symmetry, to 
the subposet induced by the top elements. The downward projection map is an 
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isomorphism from the restriction to top elements to the restriction to bottom ele- 
ments, and the upward projection map is the inverse. In the case of the congruence 
©3412, these considerations imply the following corollary of Theorem 8.2. 

Corollary 8.6. The restriction of 7rf 412 is an isomorphism from the weak order 
restricted to Baxter permutations to the weak order restricted to twisted Baxter 
permutations. The inverse isomorphism is n^ 412 . In particular, the weak order on 
Baxter permutations is a lattice. 

Recall from Section 7 that the map p is an isomorphism from the weak order 
restricted to twisted Baxter permutations to the lattice of diagonal rectangulations. 
Thus p o 7r^ 412 is an isomorphism from the weak order restricted to Baxter permu- 
tations to the lattice of diagonal rectangulations. However, the map p is constant 
on t B-classes, and thus constant on 9 3 4i2-classes, so that p o 7r| 412 = p. Thus 
we have the following corollary, where (3(R) denotes the Baxter permutation pro- 
duced from the diagonal rectangulation R by the procedure described in the proof 
of Lemma 8.4. 

Corollary 8.7. The restriction of p is an isomorphism from the weak order re- 
stricted to Baxter permutations to the lattice of diagonal rectangulations, with in- 
verse (3. 

The bijection between twisted Baxter permutations and Baxter permutations 
also allows us to define a Hopf algebra structure on Baxter permutations. Let c B 
be the map taking a Baxter permutation y to the sum, in MR, of the permutations in 
the t B-class of y. The elements of MR of the form c B {y) for Baxter permutations y 
coincide with the elements c tB (x) for twisted Baxter permutations x. As discussed 
in Section 4, these elements form a graded basis for a graded Hopf subalgebra 
of MR. Instead of realizing this Hopf subalgebra as a Hopf algebra structure on 
K[tBaxoo], we can just as well realize it as a Hopf algebra structure on K[Baxoo] = 
0„> o K[Bax„]. 

The product on Baxter permutations is necessarily x »b y = r B (c B {x) »s c B (y)), 
where r B is the linear map on K[S' 00 ] whose action on permutations is to fix Baxter 
permutations and to send all other permutations to zero. This product is written 
more simply as x» B y = r B (x» s y). In other words, x*bV is obtained by summing 
over all shifted shuffles of x and y, but ignoring those shuffles that are not Baxter 
permutations. To see why this simpler formula for the product holds, notice that 
Proposition 4.4 implies that a shuffle of two permutations x and y is not a Bax- 
ter permutation unless both x and y are Baxter permutations. (A generalization 
of Proposition 4.3 similarly accounts for the fact that the product described in 
Section 4.1 is r e (x »s y) rather than r e (c e (x) »s c e (y)).) 

The coproduct on Baxter permutations is A# = (r B ® r B ) o As o c B . Just as 
in tBax, this coproduct cannot be further simplified. Thus the coproduct Ab(x) 
of a Baxter permutation x is obtained by summing Ag(y) over all permutations y 
in the t B-class of x and then deleting all terms which contain permutations that 
arc not Baxter. 
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